shirasaya do make a temptingly attractive sword, I must admit. However, they are simple scabbard mountings and intended for nothing more deliberate.
As Triton said, the lack of tsuba or any kind of solid gripping surface is a very large drawback, ESPECIALLY when committing to any type of thrust. The shirasaya is traditionally also made of honoki. This is the same nowadays, and most western traditional shirasaya are done in poplar or alder, both of which are similarly soft and not capable of withstanding the forces directly. Real sword handles are made of this stuff too, but when given same-gawa and ito, then fitted with fuchi and kashira, you end up with something a LOT more solid. Another factor is bonding agent for the tsuka halves. Traditionally (and nowadays) it's rice glue. With more western work, wood glue is more commonly used, but neither are going to be offering much safety.
Yes, nowadays a few sayashi are working with much tougher woods like maple (or some even tougher woods if epoxy bedded to prevent resins getting all over the tang), and some are doing epoxy or acraglas beddings and bonding the tsuka together with it. If the bamboo mekugi is fitted right and the nakago has a good tight solid fit, I guess I could see it being used without severely damaging the mountings. However, for sake of grip and safety, I couldn't really recommend it.
It's a very beautiful way to see a sword displayed, but I think maybe it should stay limited to display, not use.