Except for the deserts, I'd say the middle of winter, both short term and long term, especially in higher latitudes.
Short term, the cold is, of course, the biggest ennemy. But with a good shelter and fire you can keep going a few weeks, even without food.
Long term, though, you spend much more energy producing heat, moving from point A to point B, and doing everything... and the food is very scarce. Basically you have little or no vegetal food available. If you can kill a fat cariboo you might be able to survive... but even cariboos get pretty lean by that time. So long term survival in winter is very hard. I know only one person who did survive a whole winter on his own with only a ka-bar and a small PSK, and he gets all my respect. I'm not sure I could do it.
If you can pile up some energy rich food (fat, sugar, flour) before winter, it changes many things... but this is not survival anymore. It's just living in the bush.
Cheers,
David