Besides the type and quality of your bits, it's all about the RPM and downward pressure you are exerting as you drill, not to mention the possible use of cutting oil. I used to use a type of self centering bit that didn't require a centerpunch mark, and, though it was intended mostly for sheet metal, it worked well on somewhat thicker stock, too. There are fractional size, number size, and letter size drill bits. This one was, as I recall, a #30 which worked perferctly for 1/8" rivets because it was just about .003" oversize. These bits were double ended for extended use, and they came from a supplier of sheetmetal tools and supplies. See if you have one in your area.