Trace - with all due respect - unfortunately it's not just breathing dust - dust on your skin, the fumes (formaldehyde from Micarta), solvents, etc. Maybe Bob is OK, but myself (I have an incurable liver disease that the doctors say is in part from working around solvents - many absorbed through the skin and the other part I know is from improper safety precautions on my part) and Mike Hull are a couple of examples who aren't fine due to exposure to toxic substances. Genetic propensity to such conditions is probably part of it, but I didn't know that until it was too late.
And what about cleaning up the dust? Remember it isn't just the particulates you create while you work, but the stuff that lies around the shop and gets kicked up. A lot of this stuff is cumulative and there are thousands of workers each year who become sick or worse due to long time exposure to low grade toxicity. Let's face it most of our workshops would get closed down immediately if OSHA showed up.
Finally how tough does one need? - for most ordinary (less than tactical or underwater conditions) most good dense woods will hold up just fine (frankly my first preference for a handle is antler but...).
Cocobolo is good but it can also cause health problems - I know several guys who break out in a serious rash from the dust.