Steel dust is not to bad as it is heavy and fall quickly to the floor. But Micarta, G10, carbon fiber and the like are really bad. They can stay airborn for a long time. If you have a very good collection system it is not that big of deal, but if not it "could" be a problem.
That being said there are a couple things to keep in mind. How much material are you grinding? How big is your garage? Is there anyway to limit traffic while you are grinding?
Most makers are not grinding 20 G10 handles in one sitting. And most of us work in our garages.
If you can still smell the material say 10-20 minutes after you finish, then there is still alot in the air.
Some solutions. Bandsaw as much of the material off as possible. Work with lots of ventilation. Use a collection system. Use a shop air cleaner. And work with the doors open.
Lastly, just keep in mind that this is one of the most dangerous hobby you can have if you do not take the proper safety measures.