Hi Dan,
I have drilled thousand of holes on drills ranging from bench and mag drills to 20 ton radial arms and holes from 1mm to 2 1/2 inch, the lines and center punch marks are the biggest causes of error on hole location, with incorrect uneven sharpening or dissimilar angles causing error on hole diameter, a visibly wandering bit due to the spindle or chuck is probably third. The best way to eliminate most error is to use a drill bushing on a tool steel fixture which is only viable, in my opinion, in a repetitive production environment where thousand of holes will be drilled in exactly the same place, in that scenario punching the holes would probably be more cost effective due to speed and tooling costs.
A drill wandering a bit on a pin hole in a full tang knife handle would not even enter my head as a concern, on a pivot pin in a slipjoint folder, for example, is a different matter, where 90 degree x-y-z accuracy is more critical tp the operation than the difference between a H7 and H9 reamer for controlling hole diameter. The first whack with the hammer should sort out any slack in the hole but nothing can be done if the hole is not 90 degrees to the work.
If the workpiece is clamped to the table and the position of the intended hole satisfactorily determined, the accuracy of the grind on the drill is most important for regular holes because as soon as the drill hole is at full diameter it is self centering, a case can be argued that the slacker everything is, shaft wise, the more likely the drill bit to accommodate the precision of the drill bit grind.
Disclaimer, I am not a prolific knifemaker, but my engineering knowledge, skill and experience I would consider transferable knowledge, so make of it what you will.