It might "only" be semantics, but semantics is the art/science of saying exactly what you mean.
Self identified "survivalists" generally want bad stuff to happen so they can live out their fantasies, and will probably either turn into short lived bandits or remain hidden and crying in their parents' basement if the bad stuff really does happen. They often have some borderline mental or emotional issues, and some even develop fetishistic focus on camouflage, "tacticool" gear, knives and firearms. (Just ask the guy who got *coughs* bit trying to *cough* perform an unnatural act with an autoloading shotgun and hit the bolt release!)
People who are "prepared" are ready for the nightmares to come to life, but really don't want them to happen; rather they would just like to live a quiet, ordinary, peaceful life as a productive part of society. They are actually able to function in society, and have a realistic view on the world. While considered a little out there by "normal", unprepared people (they even have a name for what we have "general anxiety disorder"), they consider us useful rather than being a danger.
I'm not a "survivalist", I'm a "preparedness geek".