Judging Criteria

[Matthew Schenker]

Everyone,
Offline, the five contest judges have been discussing the best and most fair way to pick winners for our tool contests.  A lot of great ideas have been bounced around, and I wanted to take it public so everyone can comment on it.

Remember, this is all still the beginning of the contest venture!  After we work out all the kinks in the first month, I'm confident we'll have a system we can rely on after that.

So, judges, let's post our ideas about judging!

Matthew

 

[greg mann]

Amongst the usual things like good craftsmanship, I will be looking for good narrative and description. By default, we need pictures (kind of hard to score someone without a look at the project. "Trust me, it's beautiful.", probably won't be enough.) If there are design considerations, like dealing with wood movement, they sould be explained as well, especially if there is some specific way Festools helped in that process. We seem to be heading toward a 5 category/ 1-5 scoring system. I have not decided my 5 categories and I guess WE have not decided if all our categories need to be the same.

The point is, I think a very well described and illustrated project could win out over something else even if it's made out of that waterfall bubinga that Rockler bought out of Africa a few months ago.

 

[Matthew Schenker]

Everyone,
We should have "hard" criteria, and we should be able to explain it.  At the same time, I like to also have the "soft" criteria that come from impressions and instinct.  I'm also developing a list of the five "hard" criteria for judging.

I'd be interested to hear what the other judges say on this.

I'm already putting together my short list of projects I'd like to nominate for the first month.  But it is difficult -- we have so many great projects.

One thing I'd like to add.  Judges should feel free to say something like, "I would nominate this project, if mrgoogle would just add some more information about cutting the legs."  That turns it back to the member to do a bit more.  That way, the contest becomes a method for getting members to provide more about their projects (which is kind of the bigger goal here).

Matthew

 

[Bill Wyko]

If I might suggest, a catagory for practical use and artistic use. There are some talented people here for sure. You guys have your hands full. Have fun ;D

 

[nickao]

I wasn't sure if this may be a better thread for this info:

Ideas for criteria for the contest:

  1) design integrity
  2) craftsmanship
  3) joinery
  4) aesthetics
  5) comfort
  6) stability
  7) choice of materials
  8) construction methods
  9) attention to detail
10) accuracy of reproduction
11) design functionality
12) appeal
13) pictures and explanations
14) Festool usage

Since this is an open contest with no restrictions some of these will only apply to certain projects, like stability and comfort for chairs and tables, etc. Maybe we should choose only criteria that can be applied to all projects until a time the contest focuses on one type of project.

Which are the five best for this particular contest?

Nickao

 

[Qwas]

The best Five?

In no particular order,
1. design integrity
2. design functionality
3. appeal
4. Festool usage
5. craftmanship

 

[nickao]

I think appeal(12) and aesthetics(4) may go hand and hand here. I also like choice of material(7). To me the materials can make or break an item.

Should there only be five or maybe more?

Nickao

 

[Qwas]

I think appeal(12) and aesthetics(4) may go hand and hand here. I also like choice of material(7). To me the materials can make or break an item.

Should there only be five or maybe more?

Nickao

I was trying to sway things in my favor, I'm no good at selecting wood!

 

[Qwas]

Should there be more than 5 criteria? Well you judges are going to have a tough time. And it will get tougher each month.

I think it is in the best interest of the judges to have a long list of criteria. It will be easier on the judges when the choice is down to 2 or 3 projects to have a long list.

 

[nickao]

Qwas you are so funny. 

I have a hard time picking lumber too. I have so many fantastic figured woods on hand I can not choose!

Nickao

 

[nickao]

This is a letter I sent to Matthew and I think all the judges got it. I am displaying it to everyone for comments.

Just a point about judging contests.

Usually if a winner is to be picked by each judge with no collective thinking between the judges(PM to each other) then the judging is done by a point system. As judges we all should have maybe five categories with a 1-5 point system for each category. Then the judges submit the scores and you(Matthew) tabulates them as head judge.

If a single winner is picked by a group of judges conferring, then the winner is just picked out with no point system. they work together to eliminate each submission based on their personal priorities. Other judges listen and respond back and forth and eliminate one at a time until a consensus is made as to the winner.

I am not saying we have it do it in oner of those two ways, but it is the accepted methods of judging. If you think of any sport or game show, this is ALMOST universal.

I for one think in math terms. Personally, I will write down 5 categories and look at each project and give it a value. Then after I have all the info I will look at them in relation to each other and make adjustments.

I just do not see how I could keep track of every submission otherwise. A point for you is if we should make a category sheet for us each of us so we can all look at the items in the same way and then submit them to you. Or if we just tell you our number one.

Giving points shown to the public is another issue. I think  it is probably  best to display the judging Tally's(totals only) to the public with the categories broken down, but NOT showing which judge voted how or individual judges scores, just the totals. We can not be more transparent than that. With 5 judges the numbers will speak for themselves.

Examples

Points

Boxing -  Points - judges can not discuss among themselves

Gymnastic -  Points - judges can not discuss among themselves

Dancing with the stars - Points - judges can not discuss among themselves

Conferring

American idol(before the public vote) - They confer together and just eliminate one at a time

Woodworking competition - I have seen both points and conferring

So you think you can Dance - Conferring - Points in part of the  competition, but not combined.

Does any of this make any sense?

Nickao

Matthew has already stated he does NOT want the judges conferring among themselves so a point system seems best. How the system is implemented is what Mathew is still thinking about. Suggestions can't hurt.

 

[Matthew Schenker]

Nick,
Thanks for putting together a great list of criteria for the contest!

I definitely think we can use those elements in some kind of point sytstem.  Each judge should feel free to use that system as part of his explanation of the chosen projects.  Also,  non-judges should state what they think about judging syetem.

Again, this is the first month of the contest.  We'll work out the details this month, and that should give us a good system for the future.

Matthew

 

[Brice Burrell]

  Nick, everyone, I'd like to see the judging criteria go in another direction. I feel the project itself should not be the main focus, but rather the excitement of the posters project thread. Let me give you an example, here is a link to Overtime's (Patrick) thread, Euro Cabinets 101. Patrick's thread is a "how-to" so its not meant to be a thread focused on the project. He went all out to share his techniques, photograph the process, showed us how to use some Festool and other brand tools. He put out a great deal of effort in his thread and as a result he produced one of the finest threads this forum has ever had. The fact that his finished project were only shop cabinets had no bearing on the quality of his thread. That is what I'd like the contest to be about, quality threads.

  Another issue I have with the criteria you have selected is it will give a decisive advantage to the professional woodworker, since they incorporate all of "your criteria" into their projects as their job. My idea will allow the novice to compete with the professional. What do you guys think?

I also have some very strong opinions on a points system for judging, I'll post more on that issue later.

 

[nickao]

I just started the ball rolling. The categories I posted are from several different woodworking competitions, specifically the high school competitions and I just pulled them from there.

Anything to even up the playing field sounds good to me.

I do not think we should get to far off the track, this is about woodworking and the beginners do have to strive for the same goals as a professional as far as quality, etc., If they see the categories, they can work towards them. If the categories are too tailored for a beginner they can lose sight of what they really should be trying to achieve and improve upon. This is exactly why contests are usually broken down into different skill levels and the lesser skill levels usually have slightly different categories. We do not have that luxury so we must craft a more broad set of rules.

I like Patrick's thread and would suggest just adding some categories to include that type of project, "how to" or best explanations, etc.  I am thinking that maybe more categories than 5 may be best, including some focus on what Brice is referring too.

I want the playing field even, but we must stay realistic, the novices can not have all the benefits of the scoring system. As Elji F and I were discussing in another thread, if one person has been doing this 5 or 20 years and has a "gift", lets give them the recognition due and not handicap them to much with to many rules skewed to the non-professional. All these guys want their work to be up to a pros standards anyway.

I am open to any format and  scoring(or not) system at all. So just post what you exactly propose when you get it together, Brice. And everyone can chime in on it. This is a unique contest, I see no reason why we can not craft our own unique custom set of rules.

Nickao

 

[Bill Wyko]

I only got busy with my fine woodworking after buying a mini lathe in dec2006. Does that qualify me as a novice? ;D It is only fair to mention that I've been in car audio installation for 21 years though so I've built many woofer boxes. IMHO the thread should count to a point but if the project doesn't affect the decision then why have pictures? I'm sure that as this transpires, the moderators will be fair. May I suggest that opinions I'm sure will be considered but we need to be thankful that we even have this opportunity. Maby a different critera from one month to another. i.e. best novice then best remodel then best unique use of a Festool, etc. This could open up opportunities to many different skills and some unique use of our tools as well. I think this is a great thing you're doing and I wish everyone good luck. Most importantly, lets have fun. ;D

 

[John Langevin]

I tend to agree with Brice but do not disagree with much that has been stated. As I think Matt stated  somewhere proposing this contest, I think the emphasis was to encourage higher quality, more detailed and informative postings so that each of us can learn what the poster knows/learned along the way.

I have seen many wonderful projects described here with varying amounts of detailed how-to and lessons learned. Others have posted wonderful finished projects but insufficient (imho) detail to allow me to use the same techniques. In the middle have been some wonderful posts such as Brice's on sliding dovetails that only lacked some small (but vital in my mind) details such as the selection of dovetail bits and how to easily adjust the bit parameters to set up and cut the dovetails; ie. how to set up the edge-routing guide and fine adjuster.

Overall I think the most valuable threads are those that enable the readers to actually practice and use the techniques depicted.

 

[Brice Burrell]

........Overall I think the most valuable threads are those that enable the readers to actually practice and use the techniques depicted.

This is one of the best statements in this thread.

 

[Frank Pellow]

........Overall I think the most valuable threads are those that enable the readers to actually practice and use the techniques depicted.

This is one of the best statements in this thread.

Right on!   I think that we should all attempt to live up to this objective.

 

[Daviddubya]

As the Tool Contests go from month to month perhaps each month could be a different theme.  This theme change would tend to keep interest up as well as open up the contests to new participants.  Example themes - furniture, boxes, molding and trim, Festool enhancements, shop innovations, FOG threads.

As for judging criteria, using a numerical scale is surely the way to go.  The judges could develop a list of "musts" and "wants" for each months contest.  A project is eliminated from consideration if it does not meet all of the "musts.  Each of the "wants" would be assigned a numeric value by each judge, say on a scale of 1 to 10.  The highest scoring project wins.

 

[Matthew Schenker]

As the Tool Contests go from month to month perhaps each month could be a different theme.  This theme change would tend to keep interest up as well as open up the contests to new participants.  Example themes - furniture, boxes, molding and trim, Festool enhancements, shop innovations, FOG threads.

As for judging criteria, using a numerical scale is surely the way to go.  The judges could develop a list of "musts" and "wants" for each months contest.  A project is eliminated from consideration if it does not meet all of the "musts.  Each of the "wants" would be assigned a numeric value by each judge, say on a scale of 1 to 10.  The highest scoring project wins.

Thanks David for your input.  The thing I like about our contest design is that we can move about from one subject to another.  So we could definitely go from member projects in general to something more specific.  Or we can move out of projects in certain months to consider jigs and inventions, or even tool reviews.  I see the contests as a kind of spotlight that shines on a particular area, illuminating members' creativity!  Where we shine the spotlight is open to discussion.

On the judging, it's been interesting for me so far.  The contests have given me an opportunity to look more closely at all the projects members have submitted.  We have some talented woodworkers here!

Matthew