Yes I think you're right Will.
I thought the same thing when I got my Barn Chestnut 15.
It may not be the case that Oak or American Chestnut is necessarily unfit to use for knife handles, but they certainly don't hold a candle to Ebony, African Blackwood, Lignum Vitae et al.
If I remember aright, woodturners must work African Blackwood with the same tooling that would be used for metals.
BlockPsycho, here's a link to some relative Janka scale hardness figures for timbers.
http://www.bellforestproducts.com/info/janka-hardness/
I've certainly handled old knives that had been used hard, where the ebony scales were much less battered in appearance than the brass bolsters or even the steel blade.