Ditto those who say "don't take it" -- if you really need to carry-on, those suggesting Dragonfly, small SAK, etc., are right on. If you take a multi-tool other than Micra, Crosscut, etc, pack it w/the stowed luggage.
I've been to Japan numerous times and agree w/Steve: the more formal the attire, the less hassle. I've never been stopped, and only occasionally have I seen anyone get searched.
Also, Misque's comments RE jails, "especially Japan": do you know of actual experiences? Japan is among the strictest in upholding legal restrictions, to be sure, but incarceration there is not as you suggest. Jail time would suck anywhere, but in Japan you wouldn't nearly approach the poor treatment you'd receive even in the US. If you're an American citizen, you're probably the *least* likely to face unfair treatment (although if you're not white, this drops a bit).
Probably doesn't matter either way, but are you sure you're not confusing it w/Singapore (where spitting or littering is *legally* grounds for fine or imprisonment)?