Depending on what you are doing and if the knife is some $2000 safequeen or not , you can use one of those steel 'toothbrush' in conjunction with Flitz , or steel wool , if you work at it you can usually remove the bulk of the rust , depending of course on how deep it is.
Make sure to follow the directions and buff it all off , you'll have quite a shine.