Certainly some fish are more troubled by the activity, so I would agree with rbishop, keep your movements slow until your more skittish fishes become adjusted.
As for me, all of my fish either play with or tolerate the gravel vac, except my silver dollars, who run from it like my dog runs...