There are many qualities that are considered for knife steels. Here are _some_ of them, with informal descriptions:
1. Hardness: Usually measured in Rockwell C Hardness (HRC), and is a measure of how hard it is for a small cone to make an indentation in the steel when pressed for a certain force for a certain time. For some steels, hardness can be hugely affected by the heat-treatment. Surprisingly, hardness is not well-defined in terms of fundamental physics, and instead is defined in terms of an engineering test procedure.
http://www.gordonengland.co.uk/hardness/
2. Toughness: How difficult it is to break/chip a knife. Technically, in material science, it is a measure of how much energy is needed to break a certain volume of steel. For engineering, they use a more practical method: this is typically done with a "Charpy Test" where a pendulum swings and hits a sample with specified geometry. They measure how high the starting position of the pendulum has to be, before the test sample breaks on impact. To an extent, the initial height of the pendulum represents the energy injected into the system (potential energy = mgh).
http://en.wikipedia.org/wiki/Charpy_impact_test
3. Edge Retention/Wear Resistance: How long a knife stays sharp during "normal" cutting use. Not formally defined, and measured informally for those of us without access to professional equipment. Engineers use test machines from CATRA. I don't know the details of the test, but they are something like how many layers of abrasive-impregnated paper-card a knife can cut using a standardized cutting motion which has a certain amount of force applied.
http://www.catra.org/
4. Corrosion: How resistant the metal is to rusting. Not clear to me how to evaluate this formally, but engineers sometimes soak a steel sample in acid for a set amount of time, and measure how much weight is lost compared to another steel. In practice, this is very dependent on your local environment, usage, and knife care.
5. Machinability: How difficult/easy it is to shape the metal into a knife blade (usually by cutting and grinding). If a metal is difficult to machine, then it increases costs of the knife. Often steels with qualities such as high hardness and high wear-resistance are difficult to machine. I don't know if there is a formal measure for machinability.
Many modern high-performance steels are very well optimized; so much so that they have almost all of the gains in performance that we currently know how to do (for a given process/composition and price range). This means that significant gains in one area often mean a loss of performance in another area. Not strictly true, but "sort of true" (because of how optimized they are). The hardest steels are often brittle (not tough), and some of the most corrosion resistant metals are less hard than the hardest steels, etc. Metallurgy has been progressing hugely in the last several decades, so all-around gains may be possible.
One of my favorite threads is by Ankerson about his tests of edge retention:
http://www.bladeforums.com/forums/s...based-on-Edge-Retention-cutting-5-8-quot-rope
Just keep in mind edge-retention is only one of many steel qualities.
If you are interested in material science in general, this is a great book for the general public:
http://www.amazon.com/Why-Things-Br...8834/ref=sr_1_1?ie=UTF8&qid=1317684656&sr=8-1
It's super-fun to read, and very informative; imho one of the best science books for a general audience.
The quest for knowledge is a never ending pursuit!