Yeah to add to what Frank and Bill said: It's usually quite easy to work with, but if it's really old and mineralized, it can be much more difficult. That's rarely the case though. Just use really sharp tooling. Sharp cheap AO belts are best for shaping, don't use Zirc or Ceramic belts, they're actually horrible for most handle materials, but really bad for most ivory, unless it's so old (actually fossilized, as opposed to what most of the stuff we use is, barely mineralized), that it feels like rock.
For handle work I typically use the same premium drill bits I use for steel, since I have lots of them on hand, and buy in large packs, typically the split tip style used for aggressive steel cutting. On the other hand, if I was trying to drill a long hole for a pen or something, I'd want to use premium bright finished bits used for wood or similar. Always use new bits or ones you've set aside for ivory. Trying to cheap out and use some dull as crap you've been using for everything under the sun is a guaranteed way to screw up the ivory.
If it was, as you describe; "a bear to drill" don't be surprised if stress cracks show up later. Getting ivory hot, building up stress in any way, can manifest as cracks days later.
Also, one note on the cutting/shaping issue; Personally I've found it best to avoid super aggressive belts, or super course saw blades, due to the potential for chip/tear out. I typically do my rough shaping at 60-80 grit. I've had issues with tear/chip-out when using 36 grit belts, simply because of the loose density of the grain and the size of chunks it's trying to rip out of the ivory.
I've said this before, but cheap blue AO belts that cost $2 each are what you want for handle material, keep a separate group of them that never gets used for metal, and they'll last a very long time. I buy a small handful every year or so, and use them for the same stuff repeatedly. Usually after I start feeling like they're not super sharp, then I'll finish them off doing some profiling on steel or something, but that's after many handles typically. All you need are course belts for this, and you can use the same j-flex belts at 120 and above that you use for steel for the higher grit shaping, although I still recommend keeping them separated, or at least use them first for handle work.
Many many problems people face with finishing can be attributed to dull or loaded belts.