Emre, please don't let the grain growth paranoia that inflicts most of bladesmithing deprive you of too much sleep. As you can see mete gets it about the grain size fix. Grains grow, that is what they do, that is all they do, and they never shrink. But what they do before they grow is remake themselves in a whole new phase, come to a stable size and then grow by devouring one another. This happens again, and again to varying degrees with every heat past critical (ac1 to be precise). So fixing grains is the simplest of our operations and although a number of bladesmiths promote their skills on their ability to control grain size all they are doing is preying on the public’s ignorance of this topic. Reheat your steel without going to the higher temperatures and you grains will simply reform finer with every reasonable heat. That is the good news

and now that I have you feeling better...
The bad news, the part that so many bladesmiths never seem to worry about because they are so focused on grain size, is the problem of carbide size and distribution

. This is still not too tricky to fix with a steel as simple as say 1095 as the carbon will move around rather easily during many of the same heats that can refine grain size. However, when carbide forming elements are added to the mix the carbides become very stubborn and troublesome, it is actually this reason that most bladesmiths avoid steels that are stainless or borderline stainless.
Now for the really bad news

, in D2 you have chosen about the most carbide rich stuff you can without going to stainless, about as far from 1095 as you can get without going over the top. This steel likes to form nasty course carbides that are a bear to break up. These carbides will be as much of a problem as any larger grain size.
Now for the better news. Follow proper industrial normalization concepts (even though there are none for D2) and you can fix the carbide issues

. Don't let bladesmiths talk you into low temperature cycling because of misguided ideas about grain refinement, low temp cycling will only make the carbide problem worse. Get it hot; recommended hardening temps are 980C to 1025C range, so this gives you and idea of how hot things have to be to begin putting carbide into solution. I would cycle it there a few times and then move on to grain refinement at a couple of lower temperature cycles.
The trick with this carbide stuff is how things cool more than how hot they got. Slow cooling with lots of dissolved carbide is a very dicey thing since it allows that extra carbide time to go where it wants to go, which almost never coincides where you want it to be. Your "anneal" wasn't bad because of the temperatures used alone; it was bad because of how slowly you cooled from that those temps.
For all the reason outlined above I just don't work with D2 so what I have outlined is mostly suggestions more than rules and I would pay attention to whatever mete comes up with on this as well. Also there are charts showing the marked grain reduction by going into your final quench from a less than annealed condition, no doubt by putting those carbides to good use.
P.S. Special considerations for decarb are often described for this steel, so you may have another nasty problem to be concerned with in all this heating and cooling.