Baton work is not abusive to knives, it's abusive to certain blade/edge geometries and steel types/hardnesses. There are axes that have edges thinner than most factory knives, and there are factory knives with edge angle greater than almost any cheap Home Depot axe. Call it whatever you want, it's all just steel ground down to a cutting edge.
As to the brittleness:
Factory edges involve grinding and (in some cases) laser cutting many knives in rapid succession, and it's just too easy to overheat them when you're building at production volume, especially as the edge gets thinner and thinner and loses heat capacity. Almost all stainless steel (at least the stuff that interests most modern knife enthusiasts) is highly abrasion resistant with low (compared to simpler carbon steels) machinability--making it take longer to grind, and those few extra seconds on that belt/wheel/etc., the greater the chance that you'll end up with a small section of steel that's been made file-hard and chips/breaks like glass under pressure. The good news is that, yes, once you've ground that top part off (couple hundredths of an inch), the steel underneath is usually at the right temper and improves dramatically. I've seen it over and over and over again, and not just with S30V. I had a pretty low opinion of ATS-34 once upon a time, until I'd put my own edge on it.
All of that said, no, S30V is not "tough" in the sense that most here mean it. It's highly deformation resistant, but it has very low ductility so when you do overcome its strength and force it to deform, it can't go very far and still be able to spring back. It fractures. This is not a defect of the steel, it is the NATURE of the steel. That's not a bad thing, you just have to make allowances for what it will excel at and what it won't when making your selection. Or, you go the Strider route and make the edge so thick that the properties of the steel don't show up so much. At 40 degrees included, pretty much ANY steel is tough as hell, though at that angle, none of it cuts particularly well either. I'm not singling Strider out because they're the only ones who do this, but because they use S30V and bill their knives as tough and somebody might steer the conversation that way. It's achievable, but it's achieved through highly robust geometry. Take that edge down to 25 degrees included and it'll snap, crackle and pop like any other S30V when subjected to impacts that properly done L6 at the same geometry sails through.
My knife is a:
Big Bowie---lopping, chopping, slashing, zombie beheading, heavy batoning, pig stabbing, Tonka Truck disassembly, etc. Your edge will be subjected to impact and torsional forces; needs to be able to be twisted and rocked back and forth to get it out of that knot or ribcage that it's stuck in. Properly done 5160, L6, 1084, O1, INFI and others of their ilk are your heroes.
Skinning/hunting/kitchen/box opening/carving knife---slicing, slicing, slicing, slicing and a bit of controlled push cutting (maybe you want to make some fuzz sticks). Your edge will be subjected to a high degree of abrasion and needs to be able to be accidentally pushed into hard materials like the bottom of a pan (shame on you--cut food on cutting boards!) or the bones of an animal, all without the edge wearing away too fast or blunting/flattening. S30V, 154CM, 440C, VG-10, ZDP-189 and other of their ilk are you heroes for this go around.
CAN you successfully skin with a knife made out of 5160, or chop hardwood with one made out of VG-10? Of course you can. But those aren't the applications where you'll see either steel's maximum potential. In this case, I'd recommend you stick to slicing tasks with your S30V Spyderco. More than the steel, the fact that it's a folding knife is what would keep me from trying to baton with it.
Edit to add: LOL, well, while I was writing, somebody DID bring up Strider--so there you go.