The smooth side is definitely the one you want. The best compound for the leather belt is 1 micron Boron Carbide.
I normally refine the burr right off with the leather belt as soon as I establish the primary bevel. It works very efficiently.
You can also finish with the leather. It's great for shaving or push cutting, but it's a slick edge. No tooth
That's why I finish with larger grit Cubic Boron Carbide on a linen belt.
It's unbelievable what you can accomplish with CBN on linen.
Oh, don't forget to lightly sand the smooth side of the belt with new 320 grit paper, or similar. Your belt will be smoother and hold more compound.
Just turn the machine on and hold the paper against the leather for a bit.