Tests, Reviews, Editorial Reviews & Evaluations
----------------------------------
1) A test is a process by which a candidate is expected to perform tasks in a controlled scenario with controlled equipment so as to gauge performance against an established benchmark.
2) A professional review, (which may contain test results as a sub-component) involves subject matter expertise in the form of the professional opinion of one or more experts remarking on a products efficacy both with and without the benefit of empirical test results- relying heavily on past experience.
3) Editorial reviews- these are what most consumers actually see- they range from published press releases from a company's own marketing dept. , all the way to the personal opinions of writers. Editorial reviews are mostly claims.
3) An expert evaluation is similar to a test but does not involve test results. It relies on an assessment the product , from which a forecast or prediction of efficacy or performance can be deduced, based on heuristics alone. T3 is an example
So in the case of the casual knife stress tests, they are just that, casual stress tests. Because they lack the strict controls required for emipircal data gathering, they cannot be used as a RELATIVE measure of efficacy in comparing one blade to another. They are however, much more useful to the end-user, than say an editorial review alone or even heresay.
To cite some examples of valid consumer tests:
- Consumer Reports Labs automotive and appliance tests
- IKEA's product testing labs perform measured robotic testing of same-function products against each other.
- OSHA, CPSC etc. ( agencies that offer templates for documenting test measures)
Examples of Editorial reviews are pervasive enough to not require citation. They are everything that you read on websites and in printed periodicals heralding new products or rounding up products of related type or function.These reviews are what drive new product sales.
Examples of Professional evaluations are what I do daily. By applying predictive formulae or heuristics to a range of similar products we arrive at conclusions that can be as close to delivering actionable information as lab tests- without having to actually perform lab tests. Never rely only on evaluations, or test results alone. Try to combine data from both. Consumer Reports does this well.
Why the difference matters.
If the intended (advertised) purpose of a particular blade is to sever limbs, clear vines and dismember small game- then how does it's break strength in a casual vice test reflect on it's efficacy in the real intended task? The answer is it doesn't. However, the design of the blade, it's geometry,the grip design and edge do affect the end result. So by examining these factors and alluding to heuristics from known precedents, we can- without ever taking a blade out to the woods, come to meaningful conclusions about suitability and even performance expectations.
A good evaluation involves as much data as can be gathered by a team of people- rarely just one person. So ideally, some meaningful lab test results, combined with heuristic evaluation, combined with field testing combined with long term field testing (AKA Public Beta) will deliver meaningful and actionable information that consumers can use in making buying decisions.
Far more important questions than how many chops a blade takes to sever an 8" tree or how many cinder blocks it can hack through, before failing- is weather ease of sharpening is important and if so- how easy a blade is to sharpen- how much bearing does the whole knife's ergonomics have on user fatigue, stroke effectiveness and efficiency and most importantly, longevity. It's one thing to test a 'Survival' knife's efficacy as a lever for 5 minutes and another to draw a conclusion about how well it will survive a year on a dessert island.
Lastly- there's the thorny issue of due dilligence.
Someone has to overlook tests and evaluations and act as a proctor to ensure that data isn't being fudged.
The best proctor is an unedited movie-no doubt about that. We've been making them for years and always have them available for clients to view. An underwritten and edited for brevity 'short' is also sufficient.