Sounds like your eigther forging too hot, or like you said, forging scale back into the blade, or both. If your forging at the proper temp for most carbon steel the scale will be small and light and should pop off easily, just wipe your anvil clean witha glove or rag while your re-heating the blade. One trick is to whack your blade on the side of the anvil to knock any loose scale or crap off the blade before forging.
How smooth is your anvil? the smother the better though it doesn't have to be perfect, and when forging the last few heats decrease the power of your hammer hits and "finish the blade, even out the hammer blows and smooth everything up. One trick is to put a little water on the anvil face and wet the hammer, it causes the scale to pop off the blade and makes a little cleaner forging, just wear a face shield or safety glasses and wipe the anvil clean between heats.
Also like Ed says, forge it thick, grind it then.