The hole needs to be tight but not too tight or you will damage the handle material. Some people will taper the holes so they are bigger on the outside, then they will peen the ends of the brass pins to expand them to fill the holes.
If your holes are a tiny bit larger than the pins, the space will be filled with epoxy and not matter, but depending on the handle material and level of finish the space might or might not be visible and wouldn't be acceptable in an expensive knife.