I would think that a stabilized wood handle of a suitable species would be the most durable natural handle material. The stabilization process should help eliminate any real problem with shrinking, expanding or warping. Also, some of the more exotic tropical woods hold up very well and don't seem to be bothered by climate changes.
Kevin