no hype reviews/evaluations

[Cliff Stamp]

For some time now I have been making the point that there is a lot of hype and misinformation presented as fact. Often this is intentionally to sell a product, which quite frankly is expected and something consumers have to live with. However there is also a lot of it by users and much of it not really overtly intentonal but due to things like :

-not wanting to be very critical of a popular maker/manufacturer/design

-wanting to support a friend/popular maker

-not wanting to be the first to challenge a long standing position which has become accepted as a "fact"

-letting presuppositions influence the results, the placebo effect


I have decided to become far more active in solving this problem. I can't actually stop people from spreading hype/misinformation but I can provide a source of unbiased commentary for those who are so interested. I have been discussing this with a number of makers who are of similar viewpoint and they are willing to contribute their time and effort so I will be having made a number of knives for evaluation. These will be knives which are designed purely from the viewpoint of performance.

I don't care about if they sell so they are not influenced by fads in steel or design, they are not going to be promoted by name of the maker, they are designed based on user experience, maker experience and actually supported by real *published* materials data. What I am looking for are people interested in evaluating such knives. What you get is the ability to work with extreme performance knives, including some experimental steels which have not (to my knowledge) been used in personal cutlery but have very promising materials characteristics.

However what you don't get is any information on :

-who made the knife
-what it is made from
-any details on the heat treatment

The knives will also include "defects", knives which are intentionally made wrong to provide benchmarks for errors because there is a lot of misinformation in that area as well such as a lack of second temper and then the steel chips cutting corn stalks. So you could get three knives which look identical and are simply called A,B,C. They are the same steel, one with the ideal heat treatment, one which is actually tempered in an embrittlement zone, and one which just lacks refinement (no oil/salt or cold or second temper, etc.).

This is going to test you just as much as you test the steel because you have no idea of how the steel should perform. You will have to be very critical of your method because you can't use the "accepted" behavior to see if your results make sense because there is no such information available. To be clear, I don't care what type of evaluations you do. If you just want to run working trials of paring knives in your kitchen, or you like to run controlled cardboard cutting, or you have even rigged up force probes to measure cutting ability precisely. All real and factual infomation is appreciated.

The method will be quite simple. I will have a number of blades made, I will send out an email to the evaluation group noting the type of knife, those interested will respond and the knives sent out according to the free time of the users, allow at least a month for an evaluation No one discusses the results with each other, I'll compile them into one test report. When everyone is finished, it will be disussed and be used to both evaluate the knife and the evaluation methods.

To again be very clear, I am not looking for perfection in method. Experimental science is very much a learn by doing approach. If all you learn is not how to do something then that itself is valuable because it will showcase a possible source of misinformation. The next time you correct it and the process continues.

If this sounds interesting to you then drop me an email.

-Cliff

 

[db]

Great idea I think the more users you can get the better. I really like the idea that a Bad one is in the mix.
As for the hype and bias, I really think most try to be as honest as they can when reporting on their own use and performance. I really think it is fairly rare that people lie or mislead to promote a level of performance of a knife, and really once anyone uses the same one it either works or doesn't kind of hard to hide.

 

[darkestthicket]

yeah, ill definetly break...errr test knives for you.

Also excellent post in light of specific people stating your testing is "suspect" lol.

Kudo's to you.

 

[JSR]

Cliff - sent you an email, but not sure if it went through. Let me know. Thanks.

 

[EDCeeker]

Sounds like a fascinating program, Cliff. Kudos and thanks for spear-heading it! Unfortunately, I'm in no position to participate, being as I neither use knives extensively or in hard use and I really don't know much about the finer points of knives. (Hell, I'm just figuring out how to sharpen properly.) But I look forward to the results or your and your participants' efforts.

 

[USSR]

That is a great concept. Thank you for doing this.

 

[blade14]

I was gonna tell everyone how well Dark Ops knives really do work to deflect "arterial spray" and in "deanimation operations" but I guess now I'll leave it to Cliff to test that for me. :jerkit:

 

[Cliff Stamp]

Also excellent post in light of specific people stating your testing is "suspect"

Everything is suspect, skepticism is a good thing. I have for example already did the above myself and evaluated 4 knives from Ray Kirk where I didn't know the steel or how they were hardened until after I told Ray what I have found. I was skeptical of my results long before it became a fad to do so on Bladeforums.

Unfortunately, I'm in no position to participate, being as I neither use knives extensively or in hard use and I really don't know much about the finer points of knives.

It really doesn't take a lot of skill or experience to generate useful data, it just takes a willingness to do some work.

For example, as a carpenter (or whatever) you sharpen the knives and use them at random for the same tasks on the job and as EDC. You simply record (for say a month) how often you sharpen them, how long it takes and the extent of damage if any, in as much (or as little) detail as you like.

Maybe it is found that you could not see any difference but the steels are actually fairly different. That provides a viewpoint which is valid for a lot of users and quite frankly maybe the promoted differences are not actually really significant. Maybe while those steels are different on some very controlled test you can't actually see a difference in day to day use because they are not different enough.

In fact I think a very valid question to be asked would be "How much of an advantage is actually needed for an individual to recieve a functional benefit." This will be directly answered by comparing the work done by people who want to do the very controlled work as well as the guys who just want to carry them at work and see if they are actually any better than the $5 knives they already use.

The vast majority of knives used are not in the hands of people who have Jeff Clark's sharpening skills or who can win cutting contests so if anything the work of a novice is in more representative in a lot of ways than an "expert".

One of the other things is that maybe you want to investigate the performance but are unsure of how to proceed and don't know if your ideas will work. That is the benefit of having a research group. You discuss what you plan on doing, the group comments, and ideas are refined.

-Cliff

 

[STR]

It is probably a good idea. I have mixed feelings about some parts of it.

One thing I think it can create is more controversy. Not that this is a bad thing until it gets brought out into a public format where it can be both draining, well exhausting for some, and time consuming. Many will agree that also feelings can be hurt. This of course it tough as the old saying goes but perhaps some thought should also be given to not only concealing the ID and details of the knives but also the ID and details of the evaluaters on the research group as well? At least from the makers or manufacturers of the knives. It would be fine to list all the reviewers or testers on your panel. Just don't tell which one did what maybe?

Just a thought. Some may already be involved in relationships where they do things that could be damaged by less than favorable findings, disagreements or whatever comes up. If this is contrary to what you hope to do, meaning your are really more interested in testing people so they can be nailed on a cross publically when faulted rather than test knives well, forget it.

It just seemed to me if the knives are not ID oriented why not set it up so the testers are on the 'witness protection program' as well? Perhaps the tests themselves can do all the talking this way in a way that has not been done before and you may even get some very good testers that would otherwise avoid participating due to fear of what it could do to established relationships. Something I am sure does not bother you as much Cliff as it does others. Of course I say this assuming that at some point in time the $hit will hit the fan and that you had plans to reveal those details hidden as to who made the knives. I guess as to later revealing the ID of the reviewer/tester should be up to the individual to remain anonymous or not.

STR

 

[EverTheCynic]

I think this is an excellent idea and would love to participate if I weren't deploying. Instead I'll just do my best to keep up with the results. Will your findings be put on your site or BladeForums or both? Your site is the first place where I found good reviews that put knives through their paces and I hope you get your hands on some of the knives that I would potentially buy. Break a few in half if need be. :thumbup:

 

[STR]

Reviews are not always good. One of the things that is not quite sitting well with me is that when you go into something ahead of time with the idea that hype, and misinformation is already a rampant problem in the industry that the entire exercise may turn into nothing more than some guys/testers simply getting a reputation fast for shedding a bad light on virtually half or more of the industry professionals. Something that also if true should be done but that can also destroy any chance of seeing the other half of the industries offerings should such a reputation be established. In some ways it has all the signs of somewhat of a 'witch hunt.'

It seems mandatory reading of the book, "How To Win Friends and Influence People" should be done by every member of the study group/test group or whatever panel you come up with before being allowed to participate. Changes can be made if there is a problem noted, and/or error made in stats, published data or specs or even if flat out total disinformation is discovered but it must be done in a way that tries to at least maintain relationships. It is not going to do the research team any good at all if they quickly burn down every bridge (read relationship) they ever had. My dad taught me 'never burn bridges.' You never know when you may need to cross it again. Something like this, fact based as it may be can quickly destroy otherwise good relationships Cliff. Its reality. You deal in that so I'd suggest giving thought to this side of the coin also.

STR

 

[Artfully Martial]

The primary reason for the bias (one way or another) of our knife reviews is that we are not just given random knives to review (the vast majority of the time). Instead, we carefully select from the available knives, handling them in stores, reading other's reviews etc, choosing knives with far and away the highest probability of pleasing us. That I love the vast majority of the folders I've reviewed is not a sign that I'm a pushover, only that I buy knives with a 97% certainty that I'm going to like them. This is the only rational way to do it. Indeed, the people reading the reviews will be emulating my behavior--why else would they be reading the review except to find a knife with the best probability of pleasing them?

Conversely, if we are totally certain of loving a knife (based on reviews, photos, etc) and receive it and are let down, we may have a bias in the opposite direction. If you're like me, you hype up knives to yourself and get excited about them--it's a bit like when a friend tells you that the movie you're going to see (they've already seen it) is the greatest movie ever and gets you excited about the premise. You go and see it and it turns out to be a pretty good movie--but relative to the hype, you're VERY disappointed.
Such is true of some unfavorable knife reviews (a common theme with disappointed Strider owners, I believe).

It is impossible to escape the subjectivity of something that in large part includes an artistic appreciation. You can objectively examine steel hardness, precision of machining, etc, but it's nigh impossible to objectively show that the knife handle is uncomfortable (provided it doesn't have a sharp spike sticking out or something obvious), that the knife is attractive, that the blade design is universally useful or useless, etc.

Ultimately, knives have exceeded their classification as tools and do not deserve to be reviewed as such. They don't merely serve a purpose anymore. Truth is subjectivity.

It's difficult to tell if you're discussing particular steels or entire knives (the former can probably be evaluated objectively)--you note "against the maker" etc, which would indicate the latter. What I've written becomes important or irrelevent depending on this distinction.

This sounds more like a use of Mill's methods than scientific experimentation (i.e. all blades of property x exhibit property y, but no blade that fails to exhibit property x exhibits y. Therefore, x causes y, y causes x or some other variation).

 

[hardheart]

The primary reason for the bias (one way or another) of our knife reviews is that we are not just given random knives to review (the vast majority of the time). Instead, we carefully select from the available knives, handling them in stores, reading other's reviews etc, choosing knives with far and away the highest probability of pleasing us. That I love the vast majority of the folders I've reviewed is not a sign that I'm a pushover, only that I buy knives with a 97% certainty that I'm going to like them. This is the only rational way to do it. Indeed, the people reading the reviews will be emulating my behavior--why else would they be reading the review except to find a knife with the best probability of pleasing them?

Conversely, if we are totally certain of loving a knife (based on reviews, photos, etc) and receive it and are let down, we may have a bias in the opposite direction. If you're like me, you hype up knives to yourself and get excited about them--it's a bit like when a friend tells you that the movie you're going to see (they've already seen it) is the greatest movie ever and gets you excited about the premise. You go and see it and it turns out to be a pretty good movie--but relative to the hype, you're VERY disappointed.
Such is true of some unfavorable knife reviews (a common theme with disappointed Strider owners, I believe).

But then you also face the possibility of concessions being made for any shortcomings just because of a fondness for some other aspect, the manufacturer, or some other name associated with it.

It is impossible to escape the subjectivity of something that in large part includes an artistic appreciation. You can objectively examine steel hardness, precision of machining, etc, but it's nigh impossible to objectively show that the knife handle is uncomfortable (provided it doesn't have a sharp spike sticking out or something obvious), that the knife is attractive, that the blade design is universally useful or useless, etc.

So you go ahead and evaluate the parts that you can, and leave the rest to the discretion of the individual. Reviews should not ever be designed to try and 'sell' people on a knife. You put your opinion and findings out there, and let others do what they will with it.

Ultimately, knives have exceeded their classification as tools and do not deserve to be reviewed as such. They don't merely serve a purpose anymore. Truth is subjectivity.

I couldn't disagree with this more. If knives are not tools, then we need not worry about the steel used, the heat treat performed, the integrity of the lock, the quality of the construction-the exact reasons given to us by many producers on the quality of their product, and the reason many users will suggest one model over another to someone else. Everything serves a purpose, and if part of that purpose is practical cutting application, then knives most definitely should meet some level of performance demanded by the user. When someone reviews a knife, they can tell us if it in fact did do that for them.

It's difficult to tell if you're discussing particular steels or entire knives (the former can probably be evaluated objectively)--you note "against the maker" etc, which would indicate the latter. What I've written becomes important or irrelevent depending on this distinction.

These knives are being made to Cliff's specs, and 'bad' ones are intentionally thrown into the mix. I don't know why there is concern over bad blood coming from this. If a blade breaks, you can't blame the maker when he was specifically asked to blow the heat treat. Not knowing who made it, or if it was made right, frees you up in your viewpoint. You don't go in thinking the knife 'should' perform to a certain level because someone in particular made it, or it's a special steel. You just see if it does what 'your' knife should do, not what you're told it will do.

This sounds more like a use of Mill's methods than scientific experimentation (i.e. all blades of property x exhibit property y, but no blade that fails to exhibit property x exhibits y. Therefore, x causes y, y causes x or some other variation).

But the only way to correlate the properties is to measure them. And even then, when heat treat, steel, and perhaps geometry vary among the samples, you will find different results which are affected by multiple variables. And the value of these (steel and treatment particularly) are unknown to the user. You can't determine the cause when you don't know what you are choosing from.

 

[sodak]

Great idea, I'm in. I'll send you an email.

 

[Cliff Stamp]

I guess as to later revealing the ID of the reviewer/tester should be up to the individual to remain anonymous or not.

There are a number of problems with anon commentary, but if you want to participate and pass over all your results to someone else and let them sign their name to your work then go ahead.

I think this is an excellent idea and would love to participate if I weren't deploying.

This is a long term process and assuming you can use knives over there and it is possible to get them to you then you are welcome to join.

Will your findings be put on your site or BladeForums or both?

This works as a research group meaning the individuals discuss the results among themselves and they write the final evaluation which is made public and then everyone is free to discuss it. All of the raw data is made available to the public on request but the commentary among the research group is private, not even the maker sees it.

Ultimately, knives have exceeded their classification as tools and do not deserve to be reviewed as such.

Few knives are in truth sold with no concern for aesthetics and someone else can evaluate the artistic ability of said makers. What I am interested is in how they perform, but if it isn't your primary focus then it isn't. I don't read a lot of romance novels, doesn't mean they should not be written.

In some ways it has all the signs of somewhat of a 'witch hunt.'

A witch hunt is a fairly harsh term, especially if you apply it to an investigation of this type as it implies people wil be making up problems where they don't exist and lying about their results to defame people. This is also kind of hard here since you don't know who it is you are defaming.
It is no different than the work Swaim and Talmadge did on rec.knives where they exposed a lot of myths and misinformation about knives. I am just proposing that the reviewers get held to a critical standard just like they are imposing on the knives.

These knives are being made to Cliff's specs

Sometimes it will be a maker who wants to make a point. There are issues with this of course (yeah, the one which did really well, that was the one I hardened, the other was commercial) but there are ways to deal with them also such as specific materials testing.

-Cliff

 

[EverTheCynic]

Then I may well join in in a few months when I get settled there and verify that the mail is flowing. Thank you for the opportunity.

 

[Normark]

Hey Cliff...

You and I haven't seen eye to eye on alot of things, and have even had it out in public on certain issues...

However with that being said, I think this is a Great idea and an interesting experiment.

Will be interesting to see what happens..

go for it...

ttyle

Eric
O/ST

 

[STR]

It is no different than the work Swaim and Talmadge did on rec.knives where they exposed a lot of myths and misinformation about knives

As I recall they went through some tough times about that too and even posted about the frustration of repeatedly having to defend their findings also. Well, Joe did anyway. I really don't think I've ever met or spoken with Steve. Maybe. I don't recall if so. Again though this is draining, and time consuming. I'm not totally discounting your idea Cliff. In some ways its an exciting premise.

For some time now I have been making the point that there is a lot of hype and misinformation presented as fact.

Key words here: "A lot." Perhaps 'witch hunt' is a bad choice of words on my part, but it seems you already have a set idea, or subjective view point before going into the study when it should probably be more objective. In some sense it seems you have already made up your mind. You have already said you can't do anything about it also. So why bother? What good can possibly come from it? You get the chance to say 'you are right.' I guess.

What I see as a high probability of happening is a lot of people getting angry at each other, going round and round and finally just ending up right back where you are now before you even begin if you go into this without a care in the world for who it affects or how just because you think that responding to the so called facts as you see them or read them causes this barrier of justification to solidify conclusions you have already come to. It will be refreshing to see if something good can come of it when one tests products and each other. People get nervous before tests for a reason. Being the professor giving the test must be a reassuring position. I'll pass from participating personally.

STR

 

[EDCeeker]

It really doesn't take a lot of skill or experience to generate useful data, it just takes a willingness to do some work.

For example, as a carpenter (or whatever) you sharpen the knives and use them at random for the same tasks on the job and as EDC. You simply record (for say a month) how often you sharpen them, how long it takes and the extent of damage if any, in as much (or as little) detail as you like.

That would work if I actually was a carpenter, or worked in any other job or played in any way in which a knife would be used in any serious way on a regular basis. But I do none of these things.

 

[STR]

By the way. Change can come from anger as we all know. Good luck with the idea.

STR