If you want to go as cheap as possible, the 'Zoom H2 Handy Recorder', as is it's long and actual name, is actually quite handy. Even if you do end up buying something better, it's so small you'll definitely find uses for it. It's really easy to toss in a shoulder bag (or a man purse, or just man purse, 'cause that's what a shoulder bag is, a man purse), or toss in your pocket. As well, it can always be used as a back up. It's a self contained unit, mics, power, and storage are all on board, so it's super easy to toss on a tripod and set up next to your other field recording mics.
I've never held a Zoom H4, so I can't say much about it, if someone else has one, chime in! I've also only used a {Sony PCM-M10]4 for like, an hour, so I can't say much about that, either. I think it sounded better than my H2, but I didn't have my H2 on me when I got to borrow it, so I couldn't do any A-B'ing with what I was recording.
But H2s are pretty awesome, mine goes everywhere with me. If you pick one up, you'll probably take it everywhere you go, even if you get something better, it's just so darned handy!
