In my opinion this is what you should do. I have worked with removing rust for a very long time, and used just about everything there is...
There are many things you can do about rust; Naval Jelly, CLR, Metal Finish Pad, 1000gritwet/dry paper, Breakfree, Steel Wool... The best thing to do is go to your local gun specialties store and buy a light brass bristle brush.
It is abrasive enough to remove the black oxides from your metal (even from pits within the metal) but gentle enough of an abrasive that it will not remove coatings, treatments, or paints like as a metal finishing pad would. It will leave a look of proper care on your knife or gun. The end with short bristles will enable you to only brush a very small area (3-5 cm). I would liberally apply an oil to the knife while you brush (an oil with petroleum distillates, as they are a penetrant to oxides and rust. If you do not have oil with petroleum distillates in it, K1 kerosene works about as well)
I took the black oxides from a severely pitted Cattaraugus Quartermaster 225q, now almost immaculate. The Pits went cm's into the blade, I was able to get all the black oxides from them, exposing steel with one of these brushes and naval jelly. Inho it is the best abrasive.