It never was a problem when I did it. Also, you can buy bigger goldfish at many fish stores that match the size of your goldfish better. I've never put a tiny goldfish in the tank with my large ones, but the large one I had was an 8 inch long oranda that I had for 7 years at the time. The smallest goldfish that I added to the that tank was about 2 inches long and they got along fine.
I strongly recommend quarantining the newcomer for a while before you add it to your existing fish's tank. I had never heard of quarantining when I added 2 new goldfish to my 45 gal with the big goldfish. They brought a parasite to the tank and the newcomers survived it but my 7 year old oranda didn't!