Evaluation knives and bias

[thombrogan]

I think this subforum will be safer if we assume every reviewer was given her or his knife not only free of charge, but promised a seal-blubber calzone and a fifth of Jager if the review came out in praise of the knife.

Those are my wages and if the blubber was seasoned with basil, I edit the post to make the knife more glowing. Fujiwara Corporation didn't season my seal-blubber calzone, so I insist their Tojiro brand paring knife was too thick (which it really is) instead of pretending it's thinner and somehow a superior parer to a $6 Forschner/Victorinox knife.

 

[STR]

My dad used to say to me when I'd be writing a report for something in school things like, "just stick with the facts as you see them and thats all you can do". I think the key words are, "as you see them".

I also believe we all know that two people can witness a car accident at the same time both sitting in the front seat of a car and see and report two entirely different things to the officer interviewing witnesses. My dad was a law enforcement officer at one time and stressed to us things like how to behave in the event that something serious did come up and how we need to be very careful what we report on the record because in a situation like this the interviewer more often than not reports back not so much what you said but what he thinks he heard you say, if that makes sense.

These are all good points. Facts are something to strive for in any review but even the facts can be subjective, off, misinterpretted, or simply not come across as you might think it does in your own mind when you write it. How many times has any writer thought he conveyed his thoughts clearly only to forget about it and later see it again and re-read it only to sit there and say, boy that came across confusing, jumbled and not so clear? Hey it happens. Forgive those little human tendencies. We all do it.

In my most recent review of the two lockbacks I compared to each other I tried to stick with facts as I see them, as I like to think I always do when I talk about knives but of course I have my favorites and my own opinions that shine through occasionally as we all do. When I report to someone reading my take on something that the lockbar on the Mini Manix is .154 thick or whatever it reads I am reporting what my digital caliper told me at that time. I could measure that again tomorrow and feel the need to edit if I saw something else due to a bad battery or something else. Your knife, your measurment, and your view of those so called facts could be off a bit compared to mine also, and even if you examined the exact same knife. Did I lie if you saw something different?? Or did I just report what I honestly saw with my own eyes, in my own mind when I evaluated the knife?

I think we need to learn to give the benefit of the doubt to folks because its only fair to do that and really unless you have very good reason to call foul on someone they deserve that benefit. Of course you have to take things like, "the knife is extremely comfortable to use" with a grain of salt. There is no way to really say it correctly I guess but I suppose it is more accurate to say, "I find that it is extremely comfortable to use" with a disclaimer after that informing the reader that they may or may not see similar results. (At least that is how it appears at times to me) Perhaps this is better than just spewing forth its a super comfortable knife but when someone else buys the knife and reads that its a super comfort to use and takes your word for it, and thinks the knife just sucks after using it and that it is very uncomfortable suddenly the review was biased and the author of the review is just saying good things about a knife he likes a lot or just favoriing a company with a bias when in fact all he is guilty of is not exactly expressing what he was saying by spelling it out to someone prone to jumping to conclusions.

Anyway, that is the way I feel about it but if you want to suspect that I have an ulterior motive for being here, for doing what I do, or spending time with knives thats fine. The truth is I do this out of love for the industry, for the knives, the people, and for what its worth to keep me straight and out of trouble so I'm happily married and able to live long enough to hopefully enjoy my grandchildren, which this forum and others have done for many years for me. And that is all I have to say about this thread until I think of something else to add to it.

STR

 

[Cliff Stamp]

I suppose it is more accurate to say, "I find that it is extremely comfortable to use" with a disclaimer after that informing the reader that they may or may not see similar results ...

This is clearly subjective opinion, where you would want to take care is when you were stating something presented as objective fact, or when something is presented as a generality. For example many state edge retention comparisons as if they were generalities but in fact they only hold true for a narrow subclass of work.

440C has better edge retention than 420HC for example is a statement that many people will echo strongly. But tonly the wear resistance is superior, the edge retention advantage depends on the angle used, what is cut and how and what level of sharpness needs to be sustained.

420HC has a higher edge stability than 440C which will give it superior edge retention in many cases. If you move to chopping then edge retention is also influenced by toughness as well which is also superior in 420HC.

-Cliff

 

[Thomas W]

Some of the Bladeforums folks have received knives from makers and/or companies for the purpose of testing them out to find their strengths and weaknesses. A lot of the passarounds, in the passaround forum and in private, have been conducted at great expense to various makers and manufacturers and that's been pretty cool for all involved, in my opinion.

Whether free, retail, or discount, there's always going to be a selection bias involved. It could be visceral reaction to the knife's appearance; it could be because it's from a trusted maker; or it could be from the materials used.

The past has brought me to a point of reaching out to a few members and using their expertise, enthusiasm, and time as a way to hear back positives and negatives with steels and new knives. The results for us have been quite beneficial and changes from their feedback have been made on production pieces. Of course we have been publicly ripped by Cliff for doing this type of individual casual testing, as it was done behind the scenes with the results being for our own consumption.

...asked the decent question of whether or not receiving free knives affects the objectivity of the person performing the evalutation.

I can say that the people that we got involved were used because of their objectivity, character, and straight shooting style. We also felt they were more uniters than dividers in their personalities, thus making evaluations (positive or negative) much more enjoyable.

Nonsense, that is absurd. No really Dave , nice post - I just wanted to see if it would give me some feeling of superiority to respond like a jackass. I do disagree with one thing - I don't think it takes a doctorate to tell, & that is why I wonder why so many are apparently buying into the unified cutting theory.

And from a nonscientific basis, one person regularly questions peoples motives, honesty, and/or integrity here. No one objects to that, except a very few people. Now Steven does it, and it is now an issue.

Bias is when you cannot accept any testing unless it uses your model as its basis. Bias is when you take experimental data and think it is good science to only look at one best fit line, based on a theory that you insist is based on fundamental principles, but obviously is not.

Thought I would bring this up again in case anyone missed it.

 

[OilMan]

I can say that the people that we got involved were used because of their objectivity, character, and straight shooting style. We also felt they were more uniters than dividers in their personalities, thus making evaluations (positive or negative) much more enjoyable.

So you looked for a certain type of person to do your testing but don't consider that biased?

 

[Cliff Stamp]

Of course we have been publicly ripped by Cliff for doing this type of individual casual testing, as it was done behind the scenes with the results being for our own consumption.

No, I would not so criticise private testing, it is of course standard for most R&D, often because serious problems are revealed. As an example, the public might have an adverse reaction to know that a child's car seat actually "killed" 95% of the "babies" in the first prototype. What I pointed out that if you hand pick testors, restrict what they can say publically, and further who they can lend the knives to, then you can not use what they say to represent unbiased and feedback.

I have discussed with many makers tests they have done which they thought would showcase the strengths of their knives compared to others but in fact the opposite was true or the test failed to show anything or would be too easily misinterpreted and make their knives look bad. These tests were never made public and this should not be a surprise to anyone. It is obvious that the tests released by manufacturers of their products will be highly promotional and it is very naive to think or argue otherwise.

So you looked for a certain type of person to do your testing but don't consider that biased?

In regards to judging data or method for bias the definition used in scientific work means "not representive of the population" where the population means everything, all people, all pieces of a given steel, all such heat treatements, etc. However in research there are times you want a biased sample for various reasons.

If you were evaluating a piece of equipment for examlpe, you might get the most abusive person and the most critical person to examine/use it. This feedback is not representative of the population but can still be very useful for obvious reasons. Such users can often spot problems very quickly and it is the problems that you are interested in critically in R&D.

My brother for example is very harsh on knives, he does hard work for a living and all his knives are given to him freely by me so he does not care about how much damage they take. He would be a biased sample to estimate how the knives would in general be judged by most people of course, however as an extreme bound his work is very useful. If he doesn't have a problem then I can be very confident that few people will.

On a less extreme note, Joe Talmadge is one of the more experienced lock evaluators in the online community as is Steve Harvey. Now if you had those two individuals evaluate a lock for security and strength then the knives would be held to an extremely high standard. Again this would not be representative of the population responce, but again it would be very useful to have such data as those two individuals would be able to spot many problems quickly that others would miss.

-Cliff

 

[DaveH]

I probably am biased one way or the other, but knives being free has little to do in creating that bias. I have been both pleasantly surprised a good number of times and disappointed and number of times.

I don't hold back on criticism though, I figure I own the company to be honest hoping they will improve their product.

 

[Thomas W]

So you looked for a certain type of person to do your testing but don't consider that biased?

Oilman, I didn't personally know any of the posters I reached out to. Their background, personality, habits, hygiene, schooling, personal tastes, sexual orientation, cutlery likes & dislikes were all foreign to me.

We had honest conversations about what we were looking for, and I told them to not to hold back. Of course we took into account that it was one guy with one knife or one piece of steel. We also do our own testing internally, and it is always interesting to compare the two.

Where did I say the results were not biased?

 

[straitshot]

When I write a report, evaluation, or product recommendation, I ask a disinterested party to review the "review" before general distribution. To my dismay, I sometimes find my remarks can be easily misinterpreted and even seem slanted when it's not my intention, let alone be confusing to someone not already quite knowledgeable on the subject. Mine have been generally well-received so I doubt it is lack of ability, but a lack of perspective because I assume too much at the casual reader's expense. This may be the case with some cutlery reports as well. Perhaps it's something to consider if a concern is to avoid even unintentional bias. Just a thought... Regards, ss.

 

[Thomas W]

What I pointed out that if you hand pick testors, restrict what they can say publically, and further who they can lend the knives to..
-Cliff

Again Cliff, we don't restrict what the testors have to say publically, but have kept it limited to whom they can send samples to, sorry.

Now you're referring to a specific test, and we did have blind trial that was going on with different HT's and hardnesses in regards to 13C26.
From the results (both ours and the testors) we never used any of those recipies tested, none ever hit the streets. What relevance does the results from this type of outcome have?

We are learning Cliff, if you don't like the way we learn you may look to take a different class.

then you can not use what they say to represent unbiased and feedback.

We didn't publically use what they reported back or what we found out at all, at the time it was you that couldn't let it go (and still can't).

 

[STR]

Again for the record. When a field tester, or professional in the industry is trusted with a prototype or a model not yet in production to the public the company or maker of said knife should not even have to say that they do not want it being made public or shared with someone not approved of by them. Its a given and I think also to a large extent a common knowledge courtesy to keep it under your hat.

When a new hardness that is experimental, or a new steel that is experimental or even a new knife not yet seen by but a handful of people is being talked about it deserves to be sent to someone the company can trust that will have the highest integrity and concern for keeping it private.

STR

 

[Bastid]

I wounder if this has anything to do with declaring a certain manufacturer having the sharpest knives out of the box of all, while many manufacturers were not even tested

Bias, there is no bias, just biased facts

 

[Dog of War]

Oilman, I didn't personally know any of the posters I reached out to. Their background, personality, habits, hygiene, schooling, personal tastes, sexual orientation, cutlery likes & dislikes were all foreign to me.

LOL!, just what I needed, a sinus flush with coffee.

You've got me worried now, Thomas: are we going to start seeing a lot of new models with pink and baby blue handles?

Back on topic ... even if someone were to be significantly biased by the fact that a knife or even a number of knives were given to them, I simply don't believe this would ever result in any significant commercial advantage to the company sending them out for evaluation, or even just as gratuities. It's not going to take long before others will go out and acquire the same knife(s) and realize the performance or features are not what has been claimed, if the claims are really biased.

So, while I think it's important to be aware of possible bias when reading what people post, even sometimes appropriate to draw it to others' attention, IMO it's worse than non-constructive to allege that Thomas, Sal, or anyone is trying to garner favor and support by putting knives in the hands of enthusiasts to use and perhaps report on, either on these forums or privately.

I know what you mean, but I've got the $9 and $12 Moras from Frosts and Ericksen, so, like, my frame of reference is skewed. When a $10 knife outcuts a $150 knife, it's a little eye-opening, but when an $11 knife does, too, well, it's too close to truly notice.

So, you're saying Ragnar gave you that $11 Mora?

 

[db]

No, I would not so criticise private testing, it is of course standard for most R&D, often because serious problems are revealed. As an example, the public might have an adverse reaction to know that a child's car seat actually "killed" 95% of the "babies" in the first prototype. What I pointed out that if you hand pick testors, restrict what they can say publically, and further who they can lend the knives to, then you can not use what they say to represent unbiased and feedback.

-Cliff

LOL it appears to me that you can be overly bias without even receiving, buying, or even using a knife.

 

[Cliff Stamp]

What relevance does the results from this type of outcome have?

The results of the various hardenings were reported publically on several occasions. As for how you do it (reviews/testing) all that is important is that when public statements are made that the methods are transparent. On several occasions statements were made about the work done on those knives.

It's not going to take long before others will go out and acquire the same knife(s) and realize the performance or features are not what has been claimed, if the claims are really biased.

Generally, this is why I was doing pass arounds on knives I reviewed before Bladeforums existed. However it is not the case that misinformation on steels through bias is eliminated by a counter example and all is enlightened. Just look at the rampant amount of direct contention among makers about steels and heat treating.

This can not be excused as passion or subjective opinion because the issues are properties of steel and how they are obtained and evaluated. Cashen has been speaking out with direct materials support both independent and of his own work against claims made by many makers and still all that misinformation exists.

Back to the origional point, if the contention holds, then all a maker/manufacturer would have to do is give out more knives for evaluation and thus create enough of an outspoken sample for promotion. It need not be limited to a single case. Of course this is actually a common marketing tool, it only tends to work though if the product is actually superior or you pay the people to endorse it..

-Cliff

 

[Broos]

There should be a distinction made between discussions of possible bias, and accusations of dishonest testing. Some level of bias is difficult to avoid, and to me this is completely different than accusations of dishonesty (in testing or whatever).

There are ways to avoid most bias, but there is no sure way to protect yourself from accusations that the test was fraudulent (or designed to make one knife win).

I'm biased too. I tend not to believe vague out-of-context references or name dropping made in attempt to confirm a basis in fact during a technical discussion, when an actual answer to the question at hand could have been provided just as easily.

 

[STR]

Broos, your insightful take on things is quite refreshing and in my opinion the two best posts in this thread and I want to thank you for sharing it with us.

STR

 

[thombrogan]

Cliff,

Many of those public statements shouldn't have been completely public at the time, but they came out just the same. Maybe companies whose work hasn't been co-opted without initial permission by Benchmade, Buck, SOG, and numerous overseas companies are more transparent in their dealings, but the days of inventing like crazy without seeking IP protection in the USA ended with the passing of George Carver.

 

[Cliff Stamp]

Thom, it is always the case that experiment and reporting is not a perfect process, it is more of a continuous refinement. The most important part is the willingness to accept the need for change and constant checks on methods by independent outsiders.

-Cliff

 

[CuriousOne]

No, I would not so criticise private testing, it is of course standard for most R&D, often because serious problems are revealed. As an example, the public might have an adverse reaction to know that a child's car seat actually "killed" 95% of the "babies" in the first prototype. What I pointed out that if you hand pick testors, restrict what they can say publically, and further who they can lend the knives to, then you can not use what they say to represent unbiased and feedback.

LOL it appears to me that you can be overly bias without even receiving, buying, or even using a knife.

db - What do you mean? I just don't see the connection between R&D and bias?