I hexes work well for certain fish, but you will definitely have less fish for the volume of water than you would be able to have in rectangular tank. With most fish, it is the footprint (horizontal space) it takes up rather than the height or the number of gallons that matters.
I use 10g hexes for bettas, and I have a 15gal column tank (similar to a hex in footprint but a square instead of so many corners) that I use for my 5 dwarf puffers. As you can see, 5 tiny fish in a 15 gallon and one fish in a 10 gal is much different than you could go for stocking in a rectangular tank. If your hex has a large enough footprint, like a 60 gallon, you might be able to put a schooling fish in there, but large or active fish generally shouldnt be considered for hex tanks.