In an effort to pigeon-hole my work in order to narrow my search for potential agents, I have been trawling various sites to find definitions of the various terms. To me, the terms are all, frankly, meaningless.

And yet I can't help feeling that to others they are imbued with deep nuance. Perhaps the worst offender is literary fiction.

literary fiction focuses more on style, psychological depth, and character, whereas mainstream commercial fiction (the 'pageturner' focuses more on narrative and plot

Surely this is a nonsense term? Or, if not, where do you draw the line? Surely the best thrillers (a style of genre fiction) depend on all of the above? So why are thrillers not literary? Or is there a separate genre of literary thriller fiction?

It's a real pain in the behind as I don't think my stuff is literary fiction (although it shares some of the elements of literary fiction ... like every book that has ever been written) and I think that the authors it allies most closely with are likely to be considered literary authors. This could mean one of four things:
1. The content of my book doesn't match the style of the writing and so it will be in an awkward half-way house that will be hard to sell
2. I have misunderstood the term
3. I have chosen the wrong authors to compare with
4. Er ... something else

Balls.

Another case of suck it and see, another case of more digging to be done, but has anybody found any useful guides / references?

The key term here is 'focuses more on': of course both kinds do both, and there isn't a hard line between them; it's where the balance lies that makes the call, so it can be a personal decision, and possibly one the writer's particularly ill-placed to make.

In creative terms, I would say that 'literary' is more concerned with originality of writing, thought and style, plus subtlety of plot and character, and its readers are prepared to take on something that is perhaps less superficially compelling in terms of plot (it had better be pretty compelling in other ways, though) and may need a little more careful reading to get the best out of it.

Used by authors and readers, either 'literary' or 'commercial' can be an insult, depending on whether the speaker's intellectual snobbery is inverted or the normal way up.

The trade uses the term subtly differently. They need to pigeon-hole books so as to decide how to promote them, and tend to think of 'literary' as just another genre, in which media coverage and prestige make up for lower sales. Or don't. It's the trade who range things along the axis of commercial-literary, as a simple matter of categorisation.

If a novel falls between the two stools, then I'd tend to think that either its literary or its commercial aspects aren't working properly. Beautifully written but with an unoriginal plot or banal ideas so you can't be bothered to read further? Cracking good story with twinges of originality which hamper the story without adding up to anything? Wonderful characters who don't actually do anything? These are all simplifications, but you get the idea.

Emma

These are all simplifications, but you get the idea.

Oddly, yes.