Hey JV3. Personally, I don't like Mullein, although a lot of others do. I prefer Teasel (Dipsacus sylvestris) and Jerusalem Artichoke ( Helianthus tuberosus). When you use a weedy plant like those above, it always has a softer centre and you will get the situation you describe above. BTW, I normally use a hand drill with a wood base - my favourite - Basswood (Tilia americana)
Once again, I can't tell you much about Mullein because I've only used it a few times, but it seems that the time of year you collect Teasel determines the thickness of the "hard outside wall" relative to the diameter - the thicker, the better. The longer it has been standing dead, the thinner the wall seems.
Codger is correct about the "woody outer" being the working part.
When the "hill" in the centre of the divot gets too big, causing increased resistance to the drill, use the tip of a knife to pop it out. You can also remove some of the hearth in the centre of the divot before you start drilling.
And because pictures are always good,
here's a couple of kgd (you might remember him

) getting his first hand drill coal. If you read the rest of the thread, ignore kgd's pictures and lack thereof. It seems that he deletes them after a time.
Good luck! And if you still have problems, you can always come up for a visit.
Doc
Thought I would add this - in a survival situation, except under very special circumstances, such as in the winter with an abundance of dead plants and dry air, I wouldn't waste time trying to do a hand drill fire - bow drill is the way to go - easier on you and you benefit from the mechanical advantage of the bow. Although nothing makes your day faster than a successful hand drill coal.
And one other thing: "JV3 said: i first make the drill pointy like a pencil and make a pilot hole in the base board with the tip of my knife like how i would do a bow-drill setup." Instead of making a small divot in the hearth like you do for bow drill, carve a dish shape in the hearth in which to place the drill.