if you rinsed it "numerous times" you really shouldn't have a problem. In fact, a lot of people do this to kill off harmful bacteria.
If you are putting this into a new tank (which I'm guessing most gravel goes into) I hope you cycle it... and you could fix any problem associated with the bleach if something truly goes wrong before you even put fish in.
aaron