I suggest you to try a better quality drill bits , new one not one sharpened by you . So what if brass work hardened ? It is still brass and quality HSS drill bit should drill like in butter .WIP gunstock CPM-154, vintage micarta, shadow pattern.
I had a heck of a time drilling out the brass inset. Near as I can tell, the brass dust was clogging the cutting edge of the drill bit and just rubbing instead of cutting. It was making a squealing sound similar to when I clean out a hole on a hardened piece of steel or use a reamer to slightly open up a hole on hardened steel. In discussions with others, the speculation is I work hardened the brass. One of them also spun out during the drilling.
I tried again with stainless inset. Prior to installing it, I cut some v notches into the edges and bottom to give more places for glue to bond. I was starting to experience the inability to cut and finish the hole all the way through. I switched to small bit and got the hole through then went back to the 3/32 bit.