I've done it before with mixed results. The first time I did it was about 25 years ago when I didn't know any better and nothing bad happened, other than the part looked like it has been chewed out by a rabid beaver.
The second time I did it was about 5 years ago (I definitely knew better, but I had a good reason) and the plastic I was cutting pulled the cutter out of the chuck (the helix can do that) making the cut deeper and deeper (drill chucks don't grip that great) until the cutter froze in the plastic while the drill chuck kept spinning. Nothing went flying, but it did ruin a pretty nice drill chuck.
An X Y table on a drill press is intended for positioning. If you're using it for cutting you're going to want to take shallow depths of cut. You can expect poor finish and accuracy because runout is going to be all over the place, but it is better than nothing. As already stated it is hard on the spindle and it is possible to hurt yourself with a ham handed move, so this is not a very good practice.