Because it's easy to visualize, I'll stick to describing curly (or "flamed" or "tiger-stripe" maple as luthiers sometimes call it) like in the pic I linked to above.
When the kerf sides are against the tang, the edge of the scales will match grain-to-grain. Just like it grew. So the horizontal "tiger-stripes" or "curls" in maple for instance, will be continuous across the back of the handle. With a vertical stripe (the tang) down the middle.
When the kerf sides face away from the tang, who knows if those stripes will meet up or not. Wood-grain hardly ever comes in nice clean right angles
I contour my handles a fair amount, and I've found that putting the kerf side out, you grind away anything that matches. This is with a "straight" piece of curly (

) where the ribbons are relatively evenly-spaced throughout. On quilted maple, the matching faces get less matchier (

) as soon as you cut it, birdseye can be just as "bad" that way. Because of the randomness. Burl? Spalt? Forget about it. It's not going to match if you flip it and grind it
at all.
On high-end guitars and violins and such, a carved book-matched surface, whose two sides are symmetrical on both sides of the seam, even through the different thicknesses*, is HIGHLY prized. That's because it's just plain rare for the pattern to be so consistent throughout a given thickness of wood. Plus, it looks
really cool!
*Maple tops on electric guitars are generally 1/4" - 3/8" thick. Check the net for outfits that sell large pieces of nice wood for luthiery. I'm just sayin'
When the kerf side faces the tang, the wood is in it's natural position and it will look good no matter how you carve it. The sides of the handle will almost never be a mirror image, but you're guaranteed that it will match when you look at the spine.