You could do quite a few if you are only doing the mollies and the snails. The snails and the mollies will both likely reproduce, so be prepared for that. If you don't want babies, then just get all male molliles. Google sexing mollies for some pics to compare males to females. It's actually quite easy. Males have a sharper, more pointed looking anal fin while the female's looks just like an old-fashioned fan.
You could quite easily do 10 mollies, probably more. But, I would also consider having another fish in there for a centerpiece, just for variety. A single Bolivian ram would be a good choice or a German ram, though they are a little more sensitive.