Three things:
1) Yes, work hardening can make drilling harder. Usually not enough to prevent drilling, though.
2) After forging and before any machining (cutting/drilling/milling), any forged billet should be normalized. This is what I suspect is the main problem.
3) Many steels harden to some degree even with slow cooling. You didn't list the steels in your billet, but most knife steels will be harder with heating above the Ac1 and then cooling in air.
4) If you have used the same drill bit, the first few attempts probably dulled it a lot. Re-sharpen or replace any bit that does not want to drill a hole easily in unhardened steel.