I don't think the fish really care that much, however I suggest avoiding too much white since it can be pretty reflective and make the fish feel exposed, thus stressing them a bit.
Most of the issues I hear about with gravel colors are on the human side.
Please forgive any typos. This was sent from my Samsung Galaxy S3 using the MonsterAquariaNetwok App