From an engineering POV:
1) The bearings are made for low load. They would not last long at all under the side load of a drive wheel.
2) The drive wheel would be have to be very small, That would make the belt run very slow.
3) As a drive wheel gets smaller, it has less surface contact with the belt. This lowers the friction, and causes slip. When a load was applied to the belt ( as in grinding) a 2" or smaller drive wheel would just turn while the belt stayed still.
The torque is very low on these type motors. They are intended to turn a small mass router bit at a very high speed....not deliver high torque at a low speed. Gearing one down would not help.
4) This is the biggest one - Router motors are usually in RATED HP...not delivered HP. Just because it says 3.5HP doesn't make it more powerful than a Baldor 2HP grinder motor. It has a small fraction of that power, and a ghost of the torque. Power and torque come from the number of windings, the size of the wires in the windings, and the size of the armature. In a tiny router motor these are not present.
These and many more reasons are why a 5HP shop vac or a 3HP router won't run ANY shop equipment other than the one they are installed in.