Sure, SAKs would work as a primary carry. I've sometimes gone with a SAK as my only bladed tool, and I did just fine. Of course, SAKs do have several very serious downsides compared to a good small fixed blade, which makes me always rather go for the fixed blade than the SAK, or a fixed blade and a SAK to go with that.
To me, the worst is that the ergonomics completely blow. Either you get slippery, uncomfortable plastic scales with parts of metallic tools sticking out from between them, or the same with ice cold alox scales instead. Add to that the "shape" (or lack thereof) of the "handle", and even with the large locking SAKs you get a knife that is a pain to use for prolonged periods, compared to better ones. Then there's the whole typical folding knife thing - the blades are a chore to open, and, well, they fold, so the construction is inherently weak and failure prone, although compared to a lot of other folders in the same price range, SAKs are tough. The blades themselves have edge holding that would more rightly be called "rapid edge losing", but at least they sharpen extremely quick and easy. The blades are small even for small knives, so some tasks are painfully slow with them. And then there's of course the clean up thing - working on game or even just working in dusty or otherwise dirty conditions makes your SAK so full of dust, blood and random guts and hairs that it's a hell of a pain to clean it all up later. Mind you, that's not to say that you couldn't well use SAKs as the only blade you carry to the woods - it can be done, and it's much better than carrying nothing. Personally, I value SAKs more for their other tools than the blades. Having a small screwdriver or an awl around can be really nice at times. I do like SAKs, but the fact is that they're, being folding things, a compromise, and do not at all compare to a good fixed blade.