Nice choice, bettas are beautiful fish.
A 10g tank won't cost you much more than a small 'desktop' tank - you can get 10g kits at many aquarium stores (and Wal-marts too, I believe) that are very reasonable. A tank with more space and a filter will keep your fish in much healthier conditions, as well as allowing you some room if you decide you want to add some small fish later on.
Small tanks often come without lights or use incandescents - with a very small volume of water, these bulbs can heat the tank up significantly, stressing the fish.
A fluorescent fixture runs much cooler and provides more flexibility (you can find many types of bulbs to suit different needs). If you get a kit that comes with an incandescent light fixture, you can replace the lightbulbs with full-spectrum compact fluorescents. Using fluorescent light, you can keep some low-light plants if you want to - this isn't really feasable with a tiny tank.
I think that for a little more money, a 10g is a much better buy since it allows you to do more and keep the fish in better conditions.