Hard and strong are relative.
Compared to SOFT pure iron, pearlite is HARD. It is strong compared to Brittle Martensite.
You have the basic idea, but there is no grain growth at 1200F. Grain growth starts at temperatures above Ac1 ( 1400F), and mainly gets large above 1600F. We locked in the fine grain at the quench from just above non-magnetic.
At this point we formed fine grained martensite. We do not want to go back to austenite again if our goal is making soft pearlite.
Pearlite is a lamellar arrangement of ferrite (IRON) and cementite (Iron Carbide). It forms when a eutectic mix of carbon and iron are cooled at a rate that allows separation of the two crystals.The carbides form about 10% of the mix. Any extra carbon gets tied up as more carbides.The ferrite and cementite forms in layers ( lamellae) that look like mother of pearl under a microscope - thus the name. The layers can be coarsely laid down, or finely packed.
When coarse, the carbides can be pushed around easily,and the layers disrupted with less energy. Thus, the metal is soft - allowing drilling and filing. When tightly packed,it takes alot more energy to move the arrangement. Thus, fine pearlite can mimic martensite and be hard and even somewhat brittle.
Simply:
If you cool pearlite quickly from its upper range, the layers are randomly arranged, and it is softer. If the metal is held at the upper pearlite range for long enough, and cooled slowly, everything gets laid out in nice tight layers....which are hard.
Now, what about those rock hard extra Iron Carbides? When we did the quench from austenite, we locked them into place. By leaving them alone, they can't grow any larger without going above the solution temperature again. Thus, we do a SUB-CRITICAL annealing, at which we allow the pearlite to get in its softest state without making larger carbides.
This is done at a temperature just below Ac1. We need to hold the temperature there long enough to allow the carbides to ball up with each other ,then cool it quickly enough to avoid fine pearlite.
When this state is reached, you have spheroidal annealed soft steel.
I am going to point out that again that all pearlite is not soft like butter. It can skate a file if the carbides are well distributed ( not spheroidized). It can burn up a drill if it is too fine. Avoiding both these situations is the whole point of spheroidally annealing hyper-eutectoid steel.
If we remove the extra carbon, and stay at or below the eutectoid, then there is much less problem with slow cooling the steel.
An interesting thing is that if we form austenite, and cool it at a rate fast enough to avoid forming pearlite, but stop the cooling before we would form martensite,....we get bainite. Which has some features of both martensite and pearlite. Despite some marketing publicity, bainite has little or no use in blades, as properly tempered martensite will surpass it.