Like others, I choose the handle materials by the looks. But the feel is also important, because I like to use my slipjoints a little like worry beads. There's something relaxing and satisfying about holding a nice little knife... Closed, of course.
As far as materials go, I like almost anything. Somewhat strangely, though, I don't care much for Stag, except for Amber Stag, which I love.