Hi SammyJay,
This is something I've been thinking about myself lately as it happens. When I first started writing I planned everything out quite closely. I would break prospective novels down into individual chapters, plan out each chapter, etc. As you also discovered, I tended to find that the characters had different ideas about what they wanted to do in my story, so often the initial plan would end up being redundant after so many words.
I started working on another novel some time ago and for this one all I've done is worked out mentally where the story wants to go, jotted down a paragraph or two for the first dozen or so chapters (including chapter titles and/or opening lines if they came to me) and then ploughed in. This way gives me a guide while leaving plenty of room for things to change.
I think this is largely the same as your working method - however, I'm very keen to try out Emma's columns approach - I think that could be really useful.
On a side note, I tend to write more scripts these days. I've recently found a trick that works really well for me there - I plan the story out very carefully (in the form of what might be called a 'scriptment'
which enables me to iron out any rough spots, work out what the characters need to be doing (or what they choose to be doing at that time), and basically lets me get from beginning to end. When it comes to writing the script itself it's pretty much a case of painting by numbers: the story's there, and all I need to do is put in the dialogue.
I only really mention that as, clearly, I've found two very different working methods work for the two different writing forms. It's probably because a script needs to be a little more structured, so it's easier to have a clear plan going in.
When it comes to short stories I only really write notes if I tihnk I might forget something. Otherwise I just start at the beginning and see where I end up.
cheers
JRC