This may be off the wall, but if you had a fixture below the actual surface where the material to be ground would sit and that fixture were hinged in some way it would allow you to attach some sort of lever and give you the mechanical advantage to pull the hinged fixture (to which the magnets would be permanently attached) away from both the surface your part sits on and the part itself. It would be a very rudimentary version of an on/off switch for the magnets and would allow you to clean metal particles off of the surface. Of course you'd have to have magnets strong enough to hold through whatever surface the material to be ground would sit on. It might not be the most practical or easiest method of doing so, but it would likely cost less than a large electromagnet and would solve some of those issues to some degree.
If that doesn't make sense I apologize, it's late and I have a final exam tomorrow and my head is filled with bending moments, stresses, and deflections.