brownshoe said:
For example, as you gain experience, you will get better at tasks, thus your first tests will be different than your last.
This is why you better use benchmarks. You can then check your performance on them and adjust accordingly, but even then man you better by *really* careful when you compare a current knife to a past one.
Controlling materials also leads to really poor conclusions for the reason Joe noted. They then don't actually apply to how the knife would perform if a person used it. It is trivial to get a knife for example that cuts through standard 2x4's trivially, then take it out on a real piece of wood and it gets mangled. If you are doing this then be very aware of exactly what it is that you are really testing.
If you are just looking at an aspect of cutting ability, a stock 2x4 is fine, they are however not even close to being consistent, they will vary in terms of density by massive amounts even if you get the same grade of the same type of lumber from the same place. Even the same piece of wood from the same tree will vary massively along its length.
If you really want to see the wild effects this can make live watch the Timbersports series. The last time I caught it, the organizer made a huge deal about how the stock rounds they were using were all from the exact same tree. The guys cutting them complained endless about the variation. You could see it in the chops, some axes sailed through it like butter, the others just rang in the wood (same axes, all similar guys).
One of the funniest had to be when Dave Jewett was cutting a block, again same type of stock wood, and his axe would not only not penetrate but bounced right out of the wood. He gave it a couple of chops, the axe rang hard and he stepped off it, refused to cut the block as it was too hard and would ruin his $500 axe. It was funny because I had just spent the day cutting wood and it just struck me how my brother would respond if I said that to him when we were lot clearing.
Even some materials which you would assume are consistent like hemp ropes are not. I have some ropes which cut at ~20 lbs and another roll cuts at ~30 lbs, huge differences, same brand from the same store on the same day. I started random sampling on at least always 12 different rolls to reduce these variances and wood is *WAY* more variable than that.
I bought a dozen basswood dowels for carving comparisons, same wood, same brand, same store. Some are massively harder than others, I later bought dozens more and now again random sample through the lot adding new rods as the sample gets used up, and I don't feel reasonably confident until I have cut a dozen dowels over the period couple of weeks with a few sessions.
I have done chopping comparisons on wood with two blades with hundreds of trial runs, it will eventually go normal, but man do you need a hell of a lot of trials, and I mean a lot, like dozens before you even get a solid well formed peak. You can get a rough feel with just a single chop of course, but if you really want to rank a blade the blades be prepared for a lot of work.
On top of materials you also have the influence of user sharpening. You can't just do one run like that either, if you really want reasonable confidence you have to repeat full sharpenings a few times with each blade to make sure that you are not getting a bias there. This isn't the extent of it either, it is a pretty complicated process if you want to start really being comparative. For example be aware of enviroment, doing a rope comparison in the dead of summer isn't the same as winter, chopping is even more so.
In anything so user dependent you also should be specific, so talk about how you are chopping because method will change the results from user to user, and as well be specific about things like depth of chop so the reader can estimate his strength vs yours and consider that and quantify the geometry of the knife as much as possible to eliminate a huge bias from a blade with a edge that was way thinner or thicker than the factory norm.
Before you start going into detail about what should be done, ask yourself this very important question, exactly what do I want to look at, and what precison do I want to obtain. The latter is the critical part because it determines what you need to do to get your results repeatible. Bottom line though, unless you are a maker using this to promote your work, you better enjoy it as otherwise you are not going to keep doing it for any length of time.
As for why pry, hammer, etc., because if you were not doing that then we would all be using knives which looked like Alvin's. If you are just cutting soft woods and clear materials, and never pry or bang on a knife, there is no reason to use anything but steels like M2, CMP-15V, etc., differential tempers are a waste of time, run the steels full hard, ~66 HRC, and run the edges down to nothing, with a tiny microbevel at ~10 degrees per side.
You elimate this type of work and suddenly all those tactical knives, the heavy duty knives, emergency / survival, etc., all of those have their entire scope of work cut out from under them. That is a horribly biased perspective and just as bad as taking a Mora and just doing really rough work with it and showing poor performance throughout the whole review.
The thing about durability is that it is also *WAY* harder to compare. Take two knives and cut woods and one edge fails and the other doesn't. However that same blade could have cut a hundred sections without fail, it just happened to twist the wrong way. It takes a lot of work here to get reasonable confidence. Note even then be really careful what you actually conclude, it may not be the steel's problem, it could be the geometry of the edge, or even the influence of the handle.
I have been doing this for 10 years and I am still constantly changing how I do it, and constantly updating the older reviews fixing part where the conclusions need to be adjusted because of more extensive work which change the way I interpret the older work. Some times they get harsh comments like "Hey, this method described here has some pretty serious problems such as XXX, check a more recent review such as YYY for a better idea."
Talk to the maker, see what the knife is designed to do, do that if possible or if not something similar, discuss this as well. Now look at other knives you are comparing it to, talk to those makers. Talk to the users who are buying those knives and see what they are interested in. Ask yourself are you doing a review to promote a makers work, or to actually provide information to a discriminating user.
Lastly, provide some method of feedback in your reviews. There should be some way for people to comment and for other people to read it and you should *NOT* have the ability to alter their comments, rec.knives is a great place for this but it has a low readership.
The maker should also be guaranteed the ability for a direct quote, or at least a direct link to a direct quote. Again this assumes you are interested in presenting an unbiased perspective and not just a personal soapbox. If you want to do that, that's fine, but that's a rant not a review.
-Cliff