I think confusion arises because what the book trade calls literary and what we call literary are two different, but related, things. I don't think there's
any point in trying to extend the argument back beyond Henry James, which is when there first began to be a distinction made in fiction between 'High' Art (i.e. literary) and Low/Commercial/Genre whatever. Blame the Leavises for that if you like.
Talking about the modern-day book trade, writers and readers, then, a literary novel assumes some of the following in the reader, and is written to suit:
-more than experience or education in reading fiction (i.e. greater vocabulary, range of references)
-more interest in picking up abstract ideas and themes
-a greater willingness to work out what's happening from more subtle hints
-an interest in
how it's written: if we were talking painting, the brushstrokes, as well as the subject
-this
can translate into being slower-paced, or harder to read quickly, (which is when people start being rude about it).
The book trade would add to this list:
-not obviously parkable under one or other of the genre labels - Romance, Thriller, Sci-Fi, Fantasy, Chick Lit, whatever - where it would probably sell more
-likely to win literary prizes
-likely to sell less - with some notable exceptions
-inconveniently, the author will probably write something completely different for their next book
In fact, to the book trade, 'literary' is just another genre, and has its market, towards which all cover design, marketing, publicity, sales etc. are aimed. It makes up for smaller sales by prestige on the list and in the world, and every now and then you have a smash hit which crosses over into serious sales territory -
Small Island would be an example.
To me as writer and reader 'literary' embraces all of the first list, but I would add that it has to be GOOD. Quite a lot of literary fiction isn't good, just as quite a lot of genre fiction isn't: in the book trade it's a type, not a badge of merit. In fact, my working definition of 'literary' is quite independent of genre - give me a genre, and I'll give you a Great Literary Work you could label thus. The key, to me, is originality.
Commercial fiction is often original in ideas - at least as far as psychology and perhaps theme are concerned - but less often in the actual texture of the writing and the way that all the parts are put together. Generally speaking, it remains tied to real-seeming characters, and to actions and events appropriate to the genre, because familiarity is deeply important to its appeal for readers.
My working definition of literary is a book which has originality in the ideas,
and in how they're handled,
and in the writing.
The more original in all these aspects a literary novel is, the harder it
may be to read, or at least to read really fruitfully, because we're less accustomed to decoding such things. And because some things a literary writer wants to do are honestly hard to read and make sense of, this obscurity becomes a badge of literary merit. Nick Hornby said that he didn't mind Marilyn Robertson's
Gilead being hard to read, because 'She couldn't get it done any easier'. But his contention - which I agree with to some extent - is that much
soi-disant literary fiction
could in fact be done easier, and there's no literary merit in being obscure
per se. I think that wilful obscurity is one of the reasons that some people say they're annoyed by literary fiction.
But there's highly permeable membrane between literary and commercial/genre/mass-market. What was completely literary a generation or two ago - stream-of-consciousness, say, or all-present-tense - might be the staple stuff of some quite commercial writing now. Literary writers grab and play with the conventions of thrillers, or historical romance [guilty!] or sci-fi, sometimes in a conventionally post-modern way, sometimes just because they want to explore history, or the future, or love and sex.
Which is why I might be prepared to think aloud, as above, but I think many discussions of the question (never on WW, of course!
) are tiresomely reductive and/or snobbish (and snobbery operates equally tiresomely normal way up, or inverted). It's also not very helpful to writers: you write what you write. As far as I'm concerned I really prefer the book trade's thinking: different people like different kinds of book, so let's hope someone's writing something they want.
Emma
<Added>In other words, I'm agreeing with Griff.
Saying literary fiction 'is trying to do more' is precisely not saying 'better' just 'different'.