How easily are your idlers turning without a belt in place? They should turn freely. Do you know if the wheels have a spacer/bushing between the bearings? Are you using machine bushings between your bolt/axle and the outside of the bearing races? Looks like you're binding your bearings and getting too much drag.
I buy my idlers from vwjackstraw on ebay and use machine bushings on the outsides. His wheels come with internal spacers that bear against the inner race of the bearing, and with the machine bushings, any nut or bolt tension will bear against the inner race as well. This means I can essentially tighten my wheels to my platen until the bolt strips out, and my wheel bearings will still turn freely.