Gotta ask yourself what you want to do with it.
If all you want is to slot guards, you can get by with just about anything, once you figure out the quirks and take careful finishing passes.
If you want to take on more ambitious fab projects, rip ladder patterns, or mill large damascus billets instead of grinding them, the bigger the better. If space isn't an issue, look for a bridgeport sized machine at the minimum. A no 2 sized K&T (Milwaukee), Cincinatti, Van Norman, etc, will take up roughly the same space (lower height), and are a thousand times more versatile, and rigid. With the exception of drilling, where a bridgeport style machine with quill excels. I've got a K&T No 2 Vertical(4500lbs), an MSC bridgeport clone(2500lbs), and agreement to buy a K&T No 2 Horizontal with a Universal Head.
If I had to sell one of the two I have currently, it would be the MSC in a second. Only draw back of the larger K&T is the low top spindle speed (1400rpm) on this one, but it can still run 1/8th cutters fine for slotting guards. On the flip side, it has rapid traverse and true mechanical power feeds on all axis, and can easily run 4" indexable face mills, with high feeds. It'll spit chips like nobody's business, and I'm finding myself rarely using the surface grinder anymore except for finer grit ricasso work.
That being said, I mostly make damascus, so all that may be completely irrelevant to you. You can usually pick up a good mill like this for $1-2k, depending on condition and accessories, and included tooling, which is often, a lot.