If you don't want to do it... don't.
I rarely take deposits for materials these days, and I'm real careful about delivery dates. I've built my business to the point that I have a reasonable stockpile of excellent materials on hand. If someone wants a knife with dinosaur oosic and outsourced engraving and gold inlays, they're barking up the wrong tree anyway. That's just not what I do.
I also never know when I'll need to squeeze in a custom sheath job or a quick re-handle into my schedule.
On the other hand, I'm glad that so many makers only take the jobs they like! I do accept custom orders that are outside my "comfort range" in terms of design and technique. I bid such jobs high (money-wise) and long (time-wise) to give myself room to work. It's always better to bid high and under-promise, than to come in over-budget and late.