Are you saying the inside of the sheath/scabbard is rusted? If so, and if you don't care about affecting the monetary/collector value of the sword and scabbard, there are several liquid rust removers out there that dissolve rust.
And depending on the size of the opening in the scabbard, a cleaning rod for a .22 rifle with cleaning patches might be used to get deep inside to clean it out. Or, you might make such a tool yourself by bending a wire coat hanger out. Just make sure that with whatever method you use, that the cleaning patch is well secured to the "rod" so that it doesn't come off inside the scabbard.
On the other hand, if the sword/scabbard has monetary/collector value, and you care about that, any attempts at cleaning or restoration should be done by a professional. Unfortunately, by "cleaning" the sword yourself, depending on how you did it, you may have already significantly depreciated it's value.
Of course not every sword, even an old one, is a rare and valuable collectors item. And some people don't care about resale/collector value and just want to clean the rust off their sword.
