Using the water from the smaller tank won't hurt, either. It will help prevent shock from changing water conditions.
But as has been noted above, the gravel, filter, rocks, etc. are the most important parts of the transfer.
My system involves removing all fish and most of the water to 15 gal. Rubbermaid containers, keeping the filter(s) running hanging on the sides of the container(s) (they are rectangular) during the transfer.
Then I move the gravel from the smaller to the larger tank, along with what's left of the water and some new (conditioned) water and gravel. Let that settle for an hour or so, and then gently add the water, filter(s), and, finally, the fish from the Rubbermaid containers(s).
Good luck.