Walmart sometimes carries a 10g tank for $10-15. That is for a tank only, no filter, hood, or substrate is included. You can get a sponge filter, and an air pump for around $30, or just get a cheap pump and make a sponge filter. Good instructions to make them are available online (google DIY sponge filter, or search the DIY forum here, if you're interested.
Shrimp tanks really don't need much substrate. Just enough to lightly cover the bottom is enough to support the beneficial bacteria, to keep the tank cycled. Then add some cheap plants as time goes by. That will provide more surface area for bacteria, and more places for the young shrimp to perch and hide.