In dealing with any problem, the solution is to eliminate the source of the problem.
In this case it is the brass and the wood pores. Now, you really can't do anything about the brass, if that is the metal you want to use, but the wood pores are pretty easy to deal with.
Start before the inlay is made, and fill all the pores with a hard and penetrating filler. You can flood the surface with thin CA and let it cure. Do this every day for a few days, and then sand down the surface to remove the excess CA. It should be smooth and pore free.....no pores = no place for the sanding dust to lodge. Also, an impregnated surface will be denser, and resist other contaminants and stains.
Next, after cutting the inlays, and gluing in the pieces, let it cure until all glues are completely hard. A few days will be best.
Then file, or sand using a backing block, to get the leveling started. Once all surfaces are even, use progressively finer files or dry sandpaper to take the materials to their desired finish. I like files up to a number 6 cut, as files carry away the debris ( swarf). Good quality SC paper and the 3M polishing papers ( the colored sheets) will also do an excellent job. Just avoid "rubbing" in the swarf. Have a soft cloth and regularly wipe away the swarf as you work. Using water, or polishing compounds may stain the wood, so avoid them. When the inlay has been taken down by dry sanding to over 1000 grit, or better to 2500 grit, you can carefully buff the metal with a dry buffing cloth. If you power buff, use a low speed buffer, a new soft linen buff, and only a touch of matchless white compound. DON'T overbuff brass.
Realize that highly polished brass, or any soft metal, will show every trace of a scratch every time it is touched. Wipe it off with a cloth or touch it with your fingers - it will have a million micro-scratches. You are better to leave it at 1000-2500 grit and not try to polish it. It will form a lovely patina of color and matte finish in its own time anyway.
Now, here is a tip/trick.
If you used the CA to impregnate the surface and seal it all up, that can be turned to your advantage in cleaning up any stains or dirty spots in the finishing. Use a paper towel whetted with acetone to rub away the stain. A thin layer of the CA will dissolve, and carry away the stain with it. The surface will look untouched after a wipe down. I finish all my handles that have been stabilized or CA finished with a final clean off with a well dampened ( but not soaking wet) cloth and acetone. A final buff up with a clean, dry, soft cotton cloth will make it show table ready.