I would object more for the health of the goldfish than any aggression issues.
Goldfish are not the easy keepers you remember from your childhood. They need more room (20/30 gal each in adulthood), more specialized feeding, and more attention to water quality than you might imagine. The parasite and bacterial problems alone are enormous- there is simply a whole 'nother world of diseases that you'll probably never encounter with tropicals.
Goldfish also produce huge quantities of waste, and you may not be quite ready for the amounts of water changes you'll be needing to do.
While things might look fine initially problems will occur- count on it.
Anyone who keeps fancy goldfish can tell you that they truly are best served in a goldfish only tank- no cories, no plecos, no nothing else.