What kind of vise are you using?
This is where angle lock style vises typically shine, and the more expensive vises typically work the best, but the imports aren't exactly terrible either.
I use a dead blow style mallet to "set" my parts tightly against the vise and/or parallels as I'm tightening the vise. Depending on the shape of the parts, you may have to crank down on the vise a little tighter than normal. I have a quick handle, but sometimes pull out a longer handle if I need a little more leverage. Obviously this depends on how thick/solid the part is too, but if it's really thin or flimsy, I'll usually try to clamp or glue it to a block and take lighter passes on the mill.
Also, make sure your jaw bolts are as tight as you can make them. They shouldn't wiggle or move while the vise is open.