Rob's approach to this is a good one, and is probably how he would do it. And since I already have those tools on the shelf, that is how I would do it too. But it is not going to be cheap.
To do this on the cheap, you can get used or new surplus carbide endmills on ebay for cheap. Folks use them to mill aluminum or plastic in production and sometimes throw them out after a few days when the surface finish or burrs become a problem. They're still sharp enough to remove bulk material. Then to put a decent finish on it, you can take a skim cut with a fly cutter with a sharp carbide lathe tool bit. Total tooling cost, under $50.
I'd look for a couple 1/2 carbide endmills. I'd turn them about 1200 RPM, and feed them about .003" per tooth (that's feeding fairly hard, to reduce chatter) and I'd limit the depth of cut to about .010". You should plunge in air beside the part.
The fly cutter might be about 2 1/2" dia. I'd turn it about 700 RPM and feed about 8 IPM. It should only be removing a couple thou. I'm feeding hard here because the flexibility of the fly cutter, and a Bridgeport, will cause something like smearing if you don't feed hard enough on hard material. That is also why I don't recommend a large fly cutter in this application.
After this, you're still going to need to use something like a flapper wheel on an angle grinder to smooth out the tool marks.
I have had problems with fly cutters taking a deeper cut where the cuts overlap in a similar application. I'd use a slightly negative overlap, leaving a tiny tiny hump, which is easier to remove (use a file) than "blending" the entire surrounding surface down with a flap wheel