In my experience...
-The translucent cattle, ram, and buffalo horn is probably the most susceptible to humidity/temperature shrinking and also beetles.
-Real elephant ivory is durable but also prone to checking or light cracking. I imagine warthog tusk, etc. act similarly.
-Mother of pearl can be used and survive on old knives but is very fragile. A vintage and uncracked MOP knife is not an easy thing to find.
-Jigged and smooth bone, usually cattle shin bone, can be very durable if properly oiled and treated with care. It can crack if dropped or otherwise sharply impacted.
-Stag, as in India sambar, is very hard and tough, more so than bone IMHO. Can still crack from drying and impact but I'd call it significantly stronger.
-The dense and oily woods, on the harder end of the Janka scale, will be very very durable and resilient over time. Ebony/blackwood, cocobolo, the ironwoods, rosewoods, and lignum vitae all included.
-Some of the cheaper plastic covers you will see shrinking and warping with age. But, micarta and proper acrylic have the highest durability (and arguably the least character.)
Celluloid gets an honorable mention for being a highly unstable explosive used as a traditional handle material
