It will dull, and mark it up.
If you need to for survival, don't really worry about it.
Otherwise, I would use a scraper, or the spine (if the spine is square), or a multi tool! A lot of the fire sticks come with a scraper.
It is not going to really "ruin" your knife, but will make it necessary to sharpen it, and wear faster.