Hey Mike, how's it going?
For my non-bolstered type knives I finish the handle with a blue scotch brite belt that I got from Jeff Diotte, this works very well. These scotch brite belts especially the blue one are really handy for finishing synthetic handles and also for cleaning blades up if you are going to satin finish the blade. They are expensive but last a very long time, I'm only on my second one, the first one lasted me almost a year. These were mentioned at Ed's symposium a little over a year ago as a way to speed up handle finishing.
For bolstered knives I tend to buff linen micarta as it better matches the mirror polished bolsters at least for me in my little mind it does.
Actually all you might have to do is once you've sanded to 400 grit is spray the handle with WD40, this might be the solution for you. If you've tried this and didn't get what you were looking for try the scotchbrite belt.
Take Care,
Sean