The Zoom H2 is great to get started. I also have a dedicated recorder for heavy recording sessions, use of different mics etc... but the little H2 is always in my pocket. It's a fantastic machine and a really good value for the money.
If you're interested to get in better quality on for your pocket recorder, I would recommend you a Sony PCM-M10PCM-M10/R. Anyway even if you are thinking to buy a better recorder in the future, having a little one with you all the time is great.