Oooh, that's a tricky one. Not sure of the answer. I do agree that if your external narrator doesn't step away from the characters at any other point, and start saying "Little did he know...", then you don't want to be doing it at the beginning.
This made me think - is he blaming the plot for something that's more about the story?
Do you know this TED talk by Andrew Stanton? I linked to it, via my own plot-and-story post, on Richard's thread about chapters
http://www.ted.com/talks/andrew_stanton_the_clues_to_a_great_story.html
Maybe what your agent is sensing but not articulating very well is that the first few chapters aren't making the promise that Stanton talks about: the clear promise that this story offers the reader a journey which will be worth the reader's time and effort. If the reader really gets that promise, and there are lots of reasons on that first page to feel that the story will deliver, they'll stay with you.
That's the reason for the old cliché of needing a body on page one: you can absolutely assume that everyone inside the story, and everyone reading it, really needs the problem of a dead body to be solved. So we can trust that the story will deliver the solution (and boy do some readers rage if it doesn't!)
You might also want to think about whether the very beginning is unstable enough, as that's another reason the body on page one is such a good beginning: bodies have to be dealt with, bodies maybe were killed by someone, and deaths always have consequences, and so the situation is unstable, and so something must, inevitably happen. This post of mine starts with the old problem of the waking-up-opening, but it's really about beginnings in general:
http://emmadarwin.typepad.com/thisitchofwriting/2010/06/wake-up-and-rewrite.html
and the link in that to Why Should I Bother is also about openings, which might help you to think things through:
http://emmadarwin.typepad.com/thisitchofwriting/2010/04/why-should-i-bother.html
If the body doesn't happen until chapter three, how could you make the promise and the inherent instablity of the situation stronger for the reader on page one?
Edited by EmmaD at 13:09:00 on 18 June 2013