The pH would likely level off. The water would reach an equilibrium at which the pH and KH was such that further dissolution would no longer happen, or at least not to any noticeable amount.
Taking out rocks at a time would only slow the amount of time to reach said equilibrium.
Do you have any vinegar? That's an acid, albeit dilute and weak. I don't know if it's strong enough, but it may be worth a try. It would certainly show a strong positive, but not necessarily useful in ruling them out...
I'm thinking that if it was the rocks, you should have seen an increase in KH.