Challenge Town Hall extended - Concour feedback

I would have brought this up at the town hall, except we didn't have the concour results yet so we could not have known to give this feedback. 2 things:

1. The 'opt out' score of 12 points should probably be referenced (median or mean) to the other competitors' scores, not just give a flat 12 or some other number.
2. If there was some way for a challenger to learn their 3 partial scores (innovation/execution/je-ne-sai-qua), that may really help us knuckleheads learn how to improve.

Here's my reasoning:

1. Tom mentioned that only 3-4 teams took the 'opt out' this year, but when we saw the results, 12 happened to be the worst score given in the concour portion. If you want to incentivize teams to take the opt out score it should not be an effective zero. Simply using the median or mean score from the competitors that did not opt out would solve this problem completely.
2. I honestly expected better concour scores for the Saab and PLB. But both cars did not change overall position after the concour (meaning their concour scores were very close to average). When I thought about the rules (10 for innovation, 10 for execution, 5 for je ne sai qua), I realized that both cars probably scored well for innovation, poorly for execution, and pegged the meter to full for je ne sai qua. My mistake was assuming the je ne sai qua is worth more than it is. It would be really helpful feedback to me to see what my concour score breakdown is, and I think it would really help de-mystify the scores for other teams who have been similarly stumped by their concour scores in the past.

Thoughts?

Regarding the inflated scores–you're right, and the generous judges this year threw a bit of a wrench into the concours scoring. However, we can't use the mean or median, because if we did, then low-scoring cars would take it every year. That would leave high-scoring cars to be judged, which would cause the average score to creep up over time, which would create less and less of an incentive to make a pretty car. If every car that scored less than 12 points took the mean score instead, then that would make the mean score something like 18. Next year, the bottom 2/3rds of the field would take the mean score of the remaining judged cars, which would creep up to like 22. 

Instead, we're discussing a calibration car for the judges. This would be a neutral car (probably something from one of our garages) that would be parked in front of the judges during their morning meeting. We would say "this is a 12-point car, and here is why," which should keep their scoring lower throughout the day.

I was not there.  Who Judged this year?  

I was surprised and disappointed to get one point up from the bottom after all the work I did and tied with the bone stock dirty, dented, scratched, and broken Toyota next to me.  

In reply to Justjim75 :

I got an 8.3 with my Miata and I had a cute 7 year old hamming it up with Al Unser Jr.

A car that didn't work, covered in oil and oily fingerprints, dirt, and an interior full of the desperate Orilleys bags scored 14.4 while I'm flattered I just don't get it. 

In an odd way the concours is becoming my favorite part of this event. In many ways it is the most challenging and for most of us difficult for us to adapt to. 

The mulligan I suppose I understand for those who don't care. I'll never take it I want to know how my pile of garbage does in all facets of the contest. 

One thing we could consider separating the event into 

Appearance/theme

Story/presentation 

Engineering/innovation 

Have one or two experts in each field judge the cars. A really well detailed stock miata and a homemade jet engine bolted to a ladder could end up with similar scores. Pretty well engineered well presented (everyone on the concours podium this year) will still excel however the midfield would be closer. 

 your right ,  that isn't right,  A dirty car scores no points with me, but that's me.  you have a year to ready the car, my car was 99% ready (more than some) so hopefully I CAN JUST DO DETAILS, over the next year.

 and it will prob. get washed just 'cause more than once.

Robbie said:

I would have brought this up at the town hall, except we didn't have the concour results yet so we could not have known to give this feedback. 2 things:

1. The 'opt out' score of 12 points should probably be referenced (median or mean) to the other competitors' scores, not just give a flat 12 or some other number.

I'm not a competitor and have never even been to a Challenge, but to me that's like declining to qualify for a race so your default position is mid-pack instead of in the last row.

In reply to Duke :

Right, but the opt out thing was originally developed because the judges were so over-taxed, and it was supposed to be an incentive for folks to get a mid-level score (12 out of 25 possible). GRM wanted to help reduce load on the judges, so they needed to offer an olive branch to get teams to use it. This year the olive branch was a bait-and-switch.

This is a year old, but maybe it will help some folks who are still struggling to understand the Concours. 

Perspective of a Challenge Judge

This might've been hashed out before and not chosen for good reasons, so let me know if it was, but why can't people who wish to opt out from the concours simply have their concours score excluded from their final score, instead having it made up solely from the driving events? That would eliminate the problems of creating an apparent "scoring floor" or causing runaway score inflation in the concours section.

The only problem I can immediately think of is that it would complicate the math a bit, you'd basically need to use two different calculations for those who entered the concours and those who passed on it. I think simply scaling up non-concours-entrants' other event scores to match the total number could work?

GameboyRMH said:

This might've been hashed out before and not chosen for good reasons, so let me know if it was, but why can't people who wish to opt out from the concours simply have their concours score excluded from their final score, instead having it made up solely from the driving events? That would eliminate the problems of creating an apparent "scoring floor" or causing runaway score inflation in the concours section.

The only problem I can immediately think of is that it would complicate the math a bit, you'd basically need to use two different calculations for those who entered the concours and those who passed on it. I think simply scaling up non-concours-entrants' other event scores to match the total number could work?

We're saying the same thing, basically use math to make the opt out fair-ish.

The problem that Tom stated is if you make the math too beneficial, everyone will go to it. The problem I stated is if you make the math not beneficial enough, no one will go to it.

Where is goldilocks? I do like the calibration car idea.

John Welsh:

The competitor Judges were Myself and HobieRacer.

JustJim:

The scoring is not just how clean the car in fornt us was. It included (direct from the rules): 

“Innovation” covers things like design, engineering, creativity and modifications.

“Execution” covers things like cleanliness, workmanship and attention to detail.

“Presentation” covers things like originality, theme, showmanship, team spirit, moxie, chutzpah, backstory and anything else that falls under the heading of “je ne sais quoi.”

Some cars had great Innovation, others had awesome Execution, and yet others had gobs of Presentation. Some had combinations. The concours is not just a car show, it tries to weigh/balance all of the above for roughly 40 cars to provide a sprectrum across the field. It is not as easy as it look with those three lines above.

I wish that the Do Not Judge option had been more discussed/used, but everyone wants to present thier effort and all the judges can appreciate that.

I was by far the toughest judge and did have several cars scored below average/DoNotJudge.

I will try to answer question on the results from my perspective as a judge. Also, the previous discussion Paul (SVRex) linked will cover most of the questions that arrise in this one. He wwarned me to be ready for this part of the job and I accept the challenge...

Thanks,

Steve Stafford

Robbie said:

 

Where is goldilocks? I do like the calibration car idea.

The judges (myself included) did recalibrate with respect to each other very early on in the process. I do like the calibration car as well, as my pre-judging question to the folks from Amelia Island was how do you adjust before you see any cars. There was not a satisfactory answer, because of the varied builds we were preparing to see.

Our team missed the $2017 Challenge, which was the first year of the "taking a 12" rule so I missed the reasoning or explanation for that. It still confuses me where that number came from. I'm assuming that it's just meant to be an average score, half of 25. Maybe the judges should be better reminded of that and use that as their measuring stick?

The Concours has always been a bit of a guess from a competitor's view since it's subjective and based on committee. It's hard to account for taste. I think making the scoring details more open is a bit of a double edged sword. It does provide good feedback to competitors but it also has the potential for more headaches for the GRM staff when they'd have to field complaints about someone's pride and joy having a nicer sheen than someone else who scored higher. In some ways that feedback is a moot point, since the judges are now changing every year.

The competitor Judges were Myself and HobieRacer.

Great Choices!!!!!!

Thank you both for the hard work! 

Steve or john give us some insight into the differences in say my ridiculous wagon and say stampies q45.  Both got similar concourse scores.  

He developed the car over several years, added many modifications and spruced it up with a home brew paint job.

I essentially built a hot tub on top of a stock ratty Corvette.   From a judges standpoint how did we arrive at similar scores from such divergent methods?

In reply to n8 :

The judging is not by committe, but each judge rates the cars for the categories and the results are averaged. The judges CAN discuss the ratings and compare/adjust. Generally we were all on the same scale (with mine just running lower a few points, but running the same range) for 90% of the field.

The "12" should have been at least mentioned so that new teams or teams that just did some polishing could make the decision to present or just take it easy and look over the other cars to get more/new ideas.

Some of my harsh scores comes from the fact that I see professionally builtt racecars every day of my working life. That jaded me a little knowing that some of the finer details on cars could have been cleaned up with not alot of extra work. I suggested to the 'folks in charge" that a proffesionally prepped car on display may help the competitors see the differences. It was suggested for the proposed CAM special class next year to have one to compare and that would be great. 

I really like being able to get the complete story from all the competitors, instead of just a  few in my wanderings as a competitor.

In reply to pimpm3 :

I will gather my notes and make some comments to that effect, but I will need to compile the pictures I took of MY scores first. Since I have acces to my notes and not the notes of the other judges, it will be my point of view, but as mentioned above we all wound up runnning into similar final results.

In reply to pimpm3 :

I didn’t judge your car. But I have judged in the past. 

Divergent methods are irrelevant. It’s pretty basic math. 

Each component had a maximum number of poInts. If someone hit a major home run on presentation and crapped out on the other components, they may score all the points available for presentation, but I can’t give them more. I can’t move points over from the innovation column and award more. 

So, a home run in one category with a zero in the other two could be an identical score to someone who had a mediocre effort in all categories. 

Its just math. (After a subjective opinion is rendered)

In reply to stafford1500 :

Maybe committee wasn't the best word, but averaging and discussing the scores does essentially that and that's what I meant.

By the way, this wasn't meant in any negative way as I think an average is important in subjective scoring. To say that we are anything but ecstatic with our results this year would be a lie.

stafford1500 said:

In reply to pimpm3 :

I will gather my notes and make some comments to that effect, but I will need to compile the pictures I took of MY scores first. Since I have acces to my notes and not the notes of the other judges, it will be my point of view, but as mentioned above we all wound up runnning into similar final results.

I would be really interested to know the innovation/execution/presentation breakdown of the saab ute and the PLB MR2 as well. Stampie and Pimpm3 blew both Saab and MR2 away by like 3-4 concour points.

Note on the “taking a 12” rule...

Its not the bottom. If you submit your car to judging, it IS possible to score less than a 12. Several cars did last year. 

It is intended to incentivize people who are not interested in participating in this part of the event to not waste the time of the judges and event staff. 

If your car is worth 6 points and you want to be judged, expect 6 points. Not 13. 

Some people wanted to be judged, even though they knew they would score low. No problem. 

I would welcome my Q45 being used as an example mainly cause I want to improve it even more for next time.

I haven’t read the Concours scores yet. I have no idea what anyone got. But let me give a couple examples.  This is how I would have scored them:

The parking lot build:

Innovation- 1 point. There wasn’t much innovation at all. No mods, minimal creativity.  Swap an MR2 motor into an MR2. 

Execution- 3 point. Cleanliness, execution, attention to detail were nearly non existent in this build, although they did cover a lot of things in a very brief time  

Presentation- 5 points. It pegged the scale on team spirit and back story. But the MAXIMUM allowed for this is 5 points. I’d give it 20 if I could. 

Total- 9 points.