Do not despair. I think your points are wonderful. The Wine and Cigar magazines know that these data are what helps sell their issues. People want to know the data about what they buy and use. Indeed magazines like consumer reports are built on this concept.
We need to keep the demand of the task ridiculously simple and guarantee anonymity for participants. I propose posting to the forum a sheet that can be printed out on your computer. There should be two sections to the form. First, a brief summary of volunteers demographics (e.g., age, gender, and subjects background: marital arts, military, etc.). Secondly, we could use visual analog scales (VAS) that are 10 cm. in length. Without being too technical the VAS are considered ratio scales and allow for a host of comparisons to be made using them. A subject reads the sentence above each VAS and places a mark on the 10 cm. under the sentence indicating how they feel about the topic in question. We should keep the categories "non-threatening." For example categories could include: appearence, ease of carry, construction, and over all value (price/functionality). Doing things like spine whack tests sould be eliminated intially. Using safe tests, and a guarantee of confidentiallity, will increase the pool of participants. The completed forms could be e-mailed, faxed, or mailed to one person for scoring. The data would then be entered into a statistical package like SPSS for Windows. Simple measures of mode, mean, and standard error for the categories rated could be done for each knife. The reader could "eye ball" the data to compare the ratings of one knife against another. As our database builds we could statistically compare one knife against another. We could begin the project with 10 people. As the data becomes viewed on the forum I bet others will want to join in. What do you think?
I think you are headed where we ultimately want to go. Right now though, I would be happy if we can just generate some interest.
My first objective is to measure a simple cutting test. If we can pull this off we have really suceeded in creating a new paradigm. This would be the first time that we try to test something in an organized and objective format. Surely, if we can test something simple with one knife, then we can test something simple on multiple knives. Eventually, we will get to more advanced data collection techniques.
There is additional activity in under the thread in the General Discussion Forum. You may want to monitor that thread.