Several things:
Use soft combed linen or combed cotton unstitched buffs for handles.
Don't use a buff for handles that has been used on metal...ever. Use a dedicated buff for the wood.
Get a buff rake and clean the buff before buffing a handle.
Don't load up the buff. It takes almost no compound to polish wood that is properly sanded to 1000 grit.
Buff lightly. Hard pressure creates cracks and checks, can burn the wood in a flash, and creates smears of compound on the handle.
Clean off the excess compound after buffing. Denatured alcohol is good for this. Let the handle cool down completely first, then wipe off with a paper towel damp with alcohol. After wiping down, hand buff with a soft cloth. A quick touchup on the power buff may be needed sometimes, but don't charge the wheel.