Generally, 2 - 3mm is perfect. Anything can work, but if too small, it compacts, too large and the roots can't get a good purchase. I would be happy with anything around 1/8" or so.
I have coarse sand mixed with laterite and topped with a litle gravel. It works well for the most part but I still sometimes have trouble getting plants to stay put.