There are a couple of possible explanations for your issue. First, and most likely, you got the brass too hot on the grinder and melted the epoxy. This is very easy to do, as standard epoxy's melting point is in the neighborhood of 200 F. The second is that you didn't mix your epoxy well enough in the first place. I have done both. Now when I put small spacers like that in a handle, I bond them to the wood or fiber spacer with CA, then put the whole thing on a fiber backer with G-Flex. CA has a melting point of ~500F, so much harder to mess up, although still possible. Those little pieces of metal can get hot in a hurry.
It took me while to get this stuff dialed. My advice, avoid excess metal as much as possible, the less you have to grind it down, the less heat you have to deal with. If using epoxy, I'd just shape with files rather than the grinder. If you do use the grinder, stick with new sharp belts, grind just a quick swipe at a time, and stay on the coarser side.
If you go with CA, 3M CA is the way to go. The gorilla glue is garbage, it's brittle and crumbles if you look at it funny. I haven't tried JB weld, it has a temp rating of 600F for short term so that's pretty good, I might have to give it a shot.