Well, fancies generally end up substantially smaller full grown than comets or commons do, so in a sense, they have. I know what you mean though, and there are indeed some varieties that exhibit smaller adult size than others, namely celestials, bubble eyes, tosakin, and pearlscales. Another reason they did not actively pursue such breeding projects may be cultural. In Asia, goldfish are thought to bring wealth and prosperity. To that end, they were bred to look either overtly like money (ranchu), like mythical beasts (lionheads), or like fat, happy, prosperous fish (most of the rest). Breeding them small may have seemed for all intents and purposes like breeding them for less prosperity. I'm just spitballing, but it sounds like it could make sense lol.