That's pretty normal. You can reduce the amount of loading the stone receives by just swirling the swarf with a fingertip as you work - that and do not let it dry out while you're working, Either way its not a big deal, but you will need to lap that stone periodically with another flat stone or some 400 or 600 wet/dry sand paper on a flat surface (using plenty of water). Just take a pencil and make a grid on your stone and start lapping in a figure 8 or just do small circles changing direction frequently till all the pencil is gone. Also, it might be the ink rubbing off and into your stone...