I don't really gravel vac, I only just float the vac above the surface because I have sand. But what I do with moss attached to wood is I vaucum the moss itself to take out the fish poo that gets stuck in it.
Sometimes. Sometimes the moss can hold a lot of... mulm, so I put the mouth of the vac carefully on the moss itself, using the end of the vac to pin the moss in place. I lose some, but it grows back so fast it's not a problem.
If the drift wood is not supporting any thing else and it is easy to remove and replace it back in the tank. i just remove the wood clean the tank. turn the wood upside down and gently rinse it. this way you will get most of the mulm out of the moss.