For something like that, you need an abrasive metal polishing compound or a wading cloth. On the less abrasive side there are Cape Cod Cloths and Bar Keeper's Friend. On the more aggressive side there is MetalGlo and 3M Professional. IMO you need something more aggressive as it will otherwise take forever.
If the rust itself is past minor-moderate surface oxidation and has pitted, that's much more involved and generally you are going to need something like a Dremel, a drill attachment, or a small standalone product that can buff in the fashion of a small rotary buffer, in combination with a heavy cut polish.
Another consideration is that the rust itself is possibly on other parts of the metal you cannot see or reach with the knife assembled. If that is the case, controlling the rust is tricky if the knife cannot be completely taken apart for a thorough cleaning, polishing, and proper lubing.
After getting the rust off, a good protectant will ensure the knife stays rust free. Choosing a good protectant is a frustrating process at times because quite a few protectants that are popular for whatever reason outright suck.
Mineral oil is a terrible protectant and, like Militec-1, it can sometimes expedite rusting faster than doing nothing at all.