Well, those purists that did it hundreds of years ago had the time and knew their trees. I read an interesting story in school many years ago. It described which part of a farm wagon needed to be shaped, and exactly what kind of wood was preferred for the job. All told there were 45-50 species. Even taking some embellishment on the writer's part, it showed what kind of depth our forefathers had in selecting wood.
It was mentioned in another thread on decorative handles what might be ok, and what wouldn't work at all. A couple of suggestions were spot on, the wood would be OK or fail exactly as described. I burn wood for winter heat - therefore I cut down trees, and split it. Cherry is extremely easy to work with, but I wouldn't call it durable. No doubt our pioneer ancestors used a lot of it making early furniture, it was just strong enough, looked good when finished, but even more importantly, was easily resawn, shaped, and worked.
On the other hand, I've tried hickory and hedge. Exactly the opposite. Some hedge I've had to cut in half with a chain saw because the 27 ton splitter wouldn't. It could handle the smaller pieces, tho. This year I am burning oak from a 50" tree that I cut early this spring. That will be a lot of work, but it's not very difficult on the saw or splitter when it's still a bit green. Dry is something completely different.
If you are going to experiment with ax handles, cut and let dry a number of different species. Shape some green, let the others cure out. You'll gain a lot of knowledge about the ease or difficulty of it, and you will also have a backup supply of what you need. Strangely enough, ax handles do grow on trees.