The best way to determine that would be for you to buy an API master test kit. Every system is unique, so there's no reliable rule of thumb to go by. You're right in that all the built up gunk inside the filter media stays in the water column, obviously. Filtering just collects all that for us so we can remove it easily. Test your parameters and use those results to let you know when to do water changes and filter cleaning.