The lateral force will wear the bearings much faster than drilling. A cheap HF drill press is the ticket if you need one for side-cutting tasks, so your main drill press stays tight. A hand-held drill works also although it is not as easy to be precise.
You can buy sanding drums and sleeves at many hardware stores but I have yet to find sleeves with a good-quality abrasive for grinding metal. Before I got a grinder with a small wheel I used an angle grinder and files when I needed to cut a small inside radius, then use the sanding drums in a hand drill to finish it up. Trying to take all the metal off with the drums did not seem effective to me at all, the abrasive just wasn't up to it.