One "plastic" in particular that was used on old knives, and one you might want to be wary of is celluloid. As celluloid ages and the chemical composition breaks down it releases acidic fumes, known as "out-gassing". So not only can the handles shrink, but the gasses can eat metal, including steel.
This is not to say that people shouldn't buy or collect knives with celluloid handles, after all, they already exist, and they are going to do what they're going to do regardless, but the common advice is to not store knives with celluloid handles inside of anything that can trap the gasses (best to leave them out in the open, but not in sunlight), and, don't store them in close proximity to other knives, or anything else you don't want to be affected by the gasses.
But if you're specifically collecting bone and antler (stag) handle knives then you won't have to worry about celluloid. Unless you already have some with celluloid handles.