shrinking or expanding won't have anything to do with your finishing techniques, unless you overheat it, and then you may see all sorts of weird shit a day or two later.
Take to 800 grit with sharp belts (only use AO belts for most handle material) but avoid using very course belts for initial shaping, as they can tear out pieces of even the best ivory. I usually never use anything more course than 80 grit, and I leave myself plenty of room in case of a blowout when bandsawing the profile.
How high a grit you hand rub will depend on the specific material, and how good you are at getting a true finish at X grit. 1500 is about the minimum with nice ivory, then buff with white rouge, light pressure, in all directions. Examine your finish under magnification, clean with a good clean soft cotton or microfiber cloth, with a little wd40 or mineral oil to remove any residue, make sure you don't see any lower grit scratches, and decide if you're happy with the finish, if not, take it to 2k, or 3k, or try blue and pink polishing paper before buffing again. I've found in most cases I can hand rub to 1500, then use blue and pink paper then quickly buff and get a great finish that I'm happy with.
Be very careful that you don't have a contaminated buff though, having a loose wheel anywhere near a grinder can contaminate it and you'll create scratches as you're buffing the others.
After you're happy with the finish, clean, and apply Renaissance Wax sparingly, and buff with a good clean soft cotton hand buff/rag by hand, until the streaks disappear.
There's a lot of discussion about long term care of ivories, and I've had mixed results with various methods, it really depends on the ivory. You'll have to do some research on that yourself and decide what seems applicable.