I would try washing it first. Don't rub hard because if there is ground in dirt it could scratch the horn and make more work for you.
After getting off any superficial dirt, I would use sandpaper. Start with extra fine, and if that doesn't take off enough surface material, use fine or even medium (but carefully). Once you find a good starting point, sand the handle with increasingly high grits of sandpaper until you like the result. I like to go up to 2000 grit, but if this is going to be a user you might stop at 400 grit. After sanding, you could apply a coat of Hooflex, which is very good for preserving horn and bring it to a nice luster.