Ha, I originally wrote "stop thinking about rpm" in my previous reply then I removed it as it seemed mean
STOP THINKING ABOUT RPM OF YOUR CONTACT WHEEL! (except see caveat below)
You need to decide how fast you want the belt to move. I suggest 2-3000 surface feet per minute, that is the belt travels 2000 feet in one minute. Once you decide this speed you then calculate how big your drive wheel should be.
An example,
My motor is 1725RPM. My drive wheels is 3.375 inches in diameter. I calculate the circumference, pi * diameter = 10.6 inches
10.6/12 = 0.883feet
0.883feet x 1725RPM = 1523SFPM that is, circumference x motor speed = belt speed.
You will do the reverse calculation, you will decide the speed you want the belt to move and then work back to the drive wheel diameter necessary to achieve it.
I use this speed on my grinder for all operations as I have no speed control. I think I would prefer it be a little faster if I could choose my speed.
I would love to have 1000 sfpm, 2000 and 4000 if I had step pulleys. High speed for rough work and slower for finishing.
http://vintagemachinery.org/math/sfpm.aspx
As Stacy explained your contact wheel will turn at a much slower speed. I suggest you read his explanation again. The large contact wheel will turn much slower than the smaller drive wheel (determined by the ratio of their diameters). Since you have a homemade wooden wheel this is a very good thing.
I think maybe the full weight of the large motor will be too much for the abrasive belt?