Lapis is a moderate to good material for full sized handles, comparable with jades and other stone. The big problem with lapis is that it often has veins of other materials running through it, and is succeptible to crumbling while it is being worked. This means that a maker may be required to go through several rare, very expensive large chunks before finding one that will survive the lapidary process. End result is a VERY expensive handle. This was a primary factor in choosing jade for my Broadwell dagger instead of the lapis that I had initially desired (Yeah, choosing. Dave practically pleaded with me to give it up!).
It's great for inlays, and when used in that fashion, there are even composites made of compressed material that can serve well and relatively economically.