If this is one of those diecast zinc/aluminum deals, I know what you're talking about, I have them on my grinder. They were pretty bad.
A lathe wouldn't help much you unless you wanted to sleeve the thing, because you don't want to open up your hole, you're wanting to straighten it.
It seems to me this is the sort of thing that machinists used to be able to do, but you don't find as many today that do this much.
The process of scraping away the material that shouldn't be there, and shimming or upsetting material where you need it isn't something I can explain very well in this format, but yes, it can be done with nothing more than a file or narrow knife to scrape with, and shims. The trouble is, if you don't have a reasonably good surface for your shaft to bear against when you finish, or if your work is able to rock, you may find you make your problem worse.
Mine came with quite a lot of slop. Perhaps you could drive a small wedge in a couple strategic spots and reduce your wobble. A small nail hammered out flat would be a good wedge.