for annealed steel, I drill a 1/4 hole at low speed using hss bits. Then after the heat treatment and cleaning, I end up with a hair- 128th to 64th - more diameter. Just happens that way.
(for 1/8 and 3/16 pin stock I use 1/8 and 3/16 bits)
Here's the difference. Some "quarter inch" stock is bigger than other 1/4" stock. There's also going to be a change in diameter when/if you rough the surface of the pin/tubing stock to grab the epoxy/glue. I do a light pass on 80 or 100 grit, then check the fit in the handle scales. (since I drill the tangs first and there's never a problem with those.) Depending on details, I may use a bit that 128th larger when drilling the scales. And sometimes I'll have to ream afterwards- depends on the material.
If it's too tight, I'll take another quick pass on the 100 grit. "too tight" is a bit subjective. I want to get the pin through the handle assembly with light tapping using a hammer, but not whaling. If you can spin the pin in the hole with your fingers, it is definitely too small for the hole.
On a hole smaller than 3/8 inch, on stock 1/4 inch thick or less, I centerpunch and just drill. centerdrilling or predrilling isn't really necessary with a decent quality HSS bit on any annealed carbon steel I've touched. I use WD40 as cutting fluid. It works. I have "real" cutting fluid, which I use on the lathe and mill about half the time, but overall I seem to get better results with bulk WD40.
Most store brand "super" bits- carbide, tungsten, whatever they come up with- don't last. A decent HSS bit is easy to sharpen, works, and will last a while. I have a few solid carbide bits for when I pull a brain fart and forget to drill a hole or need to move a hole after tempering.
Carbide bits tend to be brittle, so with those I only chuck into a mill collet and use on the mill. If your drill press is solid and true, it will be fine, but carbide don't wobble, it shears.
I do ream the holes where I use tubing and then flare the tubing after glue up. For pins I will sometimes peen and sometimes not. If your adhesive has some room to grab the pin because of a good roughing pattern, it doesn't seem necessary. But, if peening, a little reaming of the holes for a slight countersink works out well.