Quite the contrary, in my viewpoint. Smaller means easier to slip into your pocket and carry without a telltale buldge. Generally speaking, the smaller the better.
Around here, my Spyderco Dragonfly caused a bit of a stir when I used it to break down some heavy cardboard boxes. I like to fly under the radar. Way under.
Unfortunately, in this area, if you carry a Victorinox Classic on your keychain and actually use it from time to time, people will call you Rambo. And if you carry tools of any sort, they'll call you MacGuyver.
They simply don't carry anything like that. Cell phones, yes. PDA's, yes. SAKs, no.