Yeah, as long as you don't plan to attach them permanently, there's no reason to stop.
If you know somewhere that does nice wood work - cabinetry, restorations, etc. - you might be able to pick up cut-offs in various species. Knives don't require a lot of wood, you'd be surprised what people in other industries call scrap.
1/4" micarta is like $7 a stick online. It starts out FLAT, which is way convenient....
if there's a woodworking supply place around they almost surely sell blocks of exotic wood that don't need stabilization.
Also, the hardwood lumber section at Home Despot will have maple and mahogany, fer instance. And cut-offs out back....