Goodhart's Law Runs Rampant in Auto Industry?

Let's start with Charles Goodhart and his law. In economist speak, it's this:

"Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."

In recognizably human language, Goodhart's Law is this:

"Whenever people figure out the index you're evaluating them on, they will be incentivized to cheat and your index will stop being a useful measurement."

Some prozaic examples would be the SAT test and tax returns. The SAT test, theoretically, is supposed to measure a student's ability to think, gather and use knowledge and apply it to real world problems. However, the test actually measures how well students can answer a relatively well-known set of questions and, as such, they rightly focus on cramming and memorization. The SAT becomes something of a joke and the "best and brightest" become, in the famous words of William Deresiewicz, basically synonymous with the concept of "excellent sheep." With tax returns, the goal is theoretically to get each citizen to pay for the services they receive from the government. However, the actual index is a dizzyingly complex maze of income statements and writeoffs and thus many people who earn a lot of money are able, legally, to appear completely broke in the eyes of the IRS. This isn't a moral complaint about SAT takers or tax accountants, incidentally, it's just Goodhart's Law in action. 

I mention this in the context of the auto industry because I have noticed something. Let's compare and contrast the following:
1. 1988 Ford Mustang 5.0L V8 vs 2009 Hyundai Genesis Coupe 2.0T. Both are similar weights, with similar market positions. Both get low to mid 20's mpg (with the 5.0 occassionally capable of low 30s in perfect conditions). The Ford makes 215 hp and the Hyundai makes 210 hp. The engines are similar weights and have similar performance characteristics. The Hyundai is better on emissions tests, but also runs extremely rich whenever you give it the beans in ways the Ford never did.
2. 1996 Chevrolet Corvette LT4 vs 2020 BMW 440i. Both are similar weights, sold at a similar (relative) price point to a similar market. The BMW theoretically gets mid 20s mpg but more realistically is around 18. The LT4 theoretically gets low 20s but I've been inside one getting 32 mpg real world. The BMW  makes 322 hp while the Chevrolet makes 330 hp. They are similar weights and have similar performance characteristics. The BMW is theoretically better for emissions but runs so rich on boost you can see the smoke coming out. The Chevrolet is theoretically worse for emissions but does not do the extremely rich thing.
3. 90s station wagons vs 2010s crossovers. For example, a 95 Camry Wagon and a 2015 Tuareg. See above, with the added caveat that "utility" vehicles are held to much lower fuel economy standards than cars. 

This all makes me think that a lot of the technological progress of the last 30 years has not been for economy or even emissions, but getting better at gaming the fuel economy tests and the emissions tests. 

Am I being unfair?

In reply to DaewooOfDeath :

VW - Dieselgate /thread.  

In reply to secretariata (Forum Supporter) :

The thought had crossed my mind.

You are dead right.

In reply to DaewooOfDeath :

Yes, that's unfair.  There is a reason the newer cars go so rich (which may end up being too rich, but whatever) where the older ones don't.  And, ironically, it is because of emissions.  When you look at how often real people drive in the conditions that require that kind of enrichment, well- it doesn't happen that much.

Especially when you compare the TP emissions from each- from the Mustang to the Hyundai, the reduction is pretty massive- in all aspects of the vehicle.  Even the '96 Vette to any 2020 model is a huge drop off.  Heck, even at WOT, not considering the particulates, it's pretty likely that the 2020 models are cleaner- it's amazing how good the catalysts are even at non stoich conditions.  

We've also been doing a lot of real world work- which is how the whole VW debacle was found.  The testing I've done real world shows some pretty robust results thanks to the spread of required tests.  Not quite the real world testing that the EU does now, but the correlation is quite strong.  The biggest difference is still in the first 20 seconds.

On the emissions front I think alfadriver would have some good input here. WOT emissions is only a small portion of the total, and running rich for that little bit of time isn't such a big deal. It's the rest of the cycle that has been much more improved. For the fuel economy I think the more recent drive cycles are also a bit more aggressive, and you can easily beat them in some cars just like the old ones. 

Though the VW dieselgate example of being able to pass the test EXACTLY and doing nothing more.

In reply to alfadriver (Forum Supporter) :

So basically you're saying that in order to get the performance required from the engine at the pointy end of the graph the emissions output might spike a bit.  But if they're solving for reduced emissions for the conditions in which the car spends 99.5% of its runtime, that spikey behaviour still doesn't make the car more pollutey than they used to be?

If I'm understanding that correctly, that doesn't sound like gaming the system.  Seems like good sense to focus on reducing emissions where you can get big gains and leave the edge case alone.

In reply to wae :

Actually, the enrichment is bad for performance.  Especially if you see black smoke.

But, yes, for modern cars, going that rich isn't as bad as it looks relative to old cars doing the same thing.  Except for particulates.  And that "loophole" is in the process if being completely closed.

The only cars that get a "pass" are the super high performing cars- as the test does not stretch their abilities.  Then again, in the real world, there's pretty much no way to test those abilities, too.  

When you have a test, things are built to the test. This is why our regulators have the challenge of making good and relevant tests and you see them change over time. I do more with crash tests than with emissions, and its even true there. There are a lot of test programs that do things differently trying new scenarios to see real world performance. In the crash world, the small overlap stuff coming in comes to mind. Wide disparities were found in real world results, so engineers went back to labs and came out with new tests that made cars be built differently so they performed better in real world scenarios. 

The concern would be very real if the testing world was static, but it is reactive and dynamic. It has to be to fight the very principle you are suggesting.

Also; dieselgate was outright fraud. It was coding to specifically put it in a testing mode that was non-compliant with the regulations. It wasnt that they found a loophole, it was a blatant cheat.  VW wasnt the only one caught out either, FCA, Nissan, Mercedes, Audi/porsche.  There are other defeat device cases all the way back to the 70's for non-diesels too. 

Apexcarver said:

When you have a test, things are built to the test. This is why our regulators have the challenge of making good and relevant tests and you see them change over time. I do more with crash tests than with emissions, and its even true there. There are a lot of test programs that do things differently trying new scenarios to see real world performance. In the crash world, the small overlap stuff coming in comes to mind. Wide disparities were found in real world results, so engineers went back to labs and came out with new tests that made cars be built differently so they performed better in real world scenarios. 

 

The concern would be very real if the testing world was static, but it is reactive and dynamic. It has to be to fight the very principle you are suggesting.

Wasn't there a case in recent years where a vehicle was manufactured with crash structure on the driver's side but not the passenger side, because all of the overlap tests were done only on the driver's side? 

Found it. Now that they test both sides, that loophole has been closed

Dont forget all of the safety items and crash technology that has gone into vehicles. The fact that we have 400hp Mustangs off the lot and is every bit better in every measurable category then all other previous mustangs (except weight, looks not being measurable), and I would say this is a healthy market place.

Your argument is an entertaining one, but you cherry picked you car examples to highlight your defense.

A top of the line corvette vs a "hot" 2+2 seater midsized sedan/coupe?

I would probably prefer the newer model year in all of your examples. But I would prefer an equivalent model year of the older cars first.

In reply to Apexcarver :

Things may be built to the test, but the tests now are pretty robust in the cross section of nominal driving.  From simple slow driving, to pretty high speed- and much more aggressive than most drivers, A/C with heat, cold, etc.  Plus the availability of portable devices are so good that on road emissions testing is quite good.  That, and the trend to real world driving for certification.  

But availability of the PEMS devices makes it pretty easy to check cars in the real world, which makes finding the bad players easier.  When you look at the penalties, the odds of making a cheater car pretty remote- and I'm still mystified that VW did what they did.  We've had people almost go to jail for what they did,  The risk of going to jail for a detectable fault really does keep people in good standing.

One thing that is missed is the 50 years of correlation between the test and the improvement to the air quality.  So even if the cars are built to the test, they do result in improvement to the air.

The ONE part of real world driving that really isn't included right now looks to be closed in the very near future.  Which is good.

In reply to alfadriver (Forum Supporter) :

Totally agree with what you are saying there.  

"going rich at WOT" /= bad emissions over 100% of the vehicle's useful life...

I will say that there is very little way for ANY person to know the emissions output of a specific vehicle without some very specialized equipment. 

What about things we don't measure as 'emissions'? like waste heat output or waste noise? Have cars gotten hotter and louder because we don't call those emissions?

alfadriver (Forum Supporter) said:

In reply to DaewooOfDeath :

Yes, that's unfair.  There is a reason the newer cars go so rich (which may end up being too rich, but whatever) where the older ones don't.  And, ironically, it is because of emissions.  When you look at how often real people drive in the conditions that require that kind of enrichment, well- it doesn't happen that much.

Especially when you compare the TP emissions from each- from the Mustang to the Hyundai, the reduction is pretty massive- in all aspects of the vehicle.  Even the '96 Vette to any 2020 model is a huge drop off.  Heck, even at WOT, not considering the particulates, it's pretty likely that the 2020 models are cleaner- it's amazing how good the catalysts are even at non stoich conditions.  

We've also been doing a lot of real world work- which is how the whole VW debacle was found.  The testing I've done real world shows some pretty robust results thanks to the spread of required tests.  Not quite the real world testing that the EU does now, but the correlation is quite strong.  The biggest difference is still in the first 20 seconds.

So two questions then:

1. Are particulates less of a big deal than I think?

2. I didn't just mention emissions, I also mentioned fuel mileage. EVs are an exception, but in many cases fuel economy is getting worse real world. The stuff about holding "utility" vehicles to low standards and thus pushing manufacturers to replace our sedans and station wagons with Escalades and X3s is certainly part of this. 

https://www.washingtonpost.com/blogs/ezra-klein/post/cafe-loophole-could-lead-to-bigger-cars/2011/12/14/gIQA3bGLuO_blog.html

I didn't harp on it because I was trying to isolate engine design, but if we look at stuff with similar markets and designs: 

1. EG Civic with a D15 will out economy almost any commuter car available today, including the expensive hybrid stuff.

2. A 3.8 Buick from 1998 will out economy practically any 200 hp family car available today.

3. Even my mom's ratty 92 F150 out economies my dad's ecoboost F150.

Btw, I'm not trying to indict you or manufacturers. If the EPA really has figured out effective techniques to reduce Goodhart behavior I am very intetested for selfish reasons. Philosophy and Education is my PhD project and I'd like to get into policy. Education, particularly evaluation stuff, is absolutely plagued with Goodhart effects and I am not above stealing from the EPA.

Robbie (Forum Supporter)

Have cars gotten hotter and louder because we don't call those emissions?

 

https://youtu.be/64YzCty-27Y

Like this?

RossD said:

Your argument is an entertaining one, but you cherry picked you car examples to highlight your defense.

A top of the line corvette vs a "hot" 2+2 seater midsized sedan/coupe?

My criteria was as follows:

1. Cars were approximately the same size and weight.

2. Cars were inflation adjusted to similar price points.

3. Cars made similar power and torque.

4. Engines were similar weight and size, because it's easy to be efficient when you don't care about packaging. 

I could have chosen "bad examples" but it would have been stuff like "anything Italian and exotic from 1995" vs "literally anything else from 2017."

No particulates are a big deal- just the limits are different.  I would suggest that if there's actual, visible smoke, something is not as correct as it should be.  Especially if it is noted in reasonably common driving- which would deposit a nice black streak on the rear of the car.  Maybe you picked out some examples where they just go too rich.

In terms of fuel economy, you may have the perception that it's getting worse, but real world measurements having it get better.  And it's easy to find a few examples of old cars that were really good- they were probably that way to offset a really bad car, at the expense of performance.  Your mom's F150 can't do what you dads can do, by any measure other than maybe fuel economy in certain driving styles.  On an entire fleet average, fuel economy is better.  Not as improved as performance or emissions- but customer demand wants performance first, and regulatory agencies understand that keeping people alive in the short term is really important.  It's taken a lot of change to raise FE enough to compromise performance- but that is coming.

I've seen the analysis that the change from cars/LDT and MDV/HDV fuel economy has changed what is being built.  But ignoring the consumer trend to buy SUV's for more money than cars misses a lot- that's been happening for decades, and I think it sucks.  Sucks bad, as I want a car and not a SUV.  But that trend is real.  And the analysis also missed the issue that FE requirements are still getting tighter, in spite of the "loophole".  

Feel free to tell me to leave, but I think some of the examples you are going for are not quite equivalent. The economy cars of past years dont get sold like that anymore. They all have more tech, more safety and more performance. Current economy cars (in the sense of MPG) are lightyears past any of those. Even an old Prius will out MPG anything designed to do the same thing from decades past while being quieter, faster, more comfortable, safer and cleaner.

Trucks are more powerful, larger, safer and more comfortable. I dont know off the top of my head but Ill bet the towing numbers on the ecoboost are significantly more than the older truck. Apart from that, Ill bet the experience is totally different too. Ive towed with old trucks. Technically doing it and comfortably doing it are different experiences.

Yes average MPG is probably stagnant from a 30k foot view. The SUV trend is part of that, large trucks and SUVs are the worst offenders. Pretty sure suburbans are still as heavy and inefficient as they have ever been. That said, my Crosstrek averages 30 while being safer, more comfortable and powering all 4 wheels at speeds much faster than the 55 mph limit of the 70s-80s-whatever. Progress has been made. And its labled as PZEV, so its cleaner than anything else I own too.

edit: Alfa was faster. I believe him that across the board MPG is up. So thats happening while all of these other improvements are too. Thats remarkable.

In reply to alfadriver (Forum Supporter) :

Btw, I'm not trying to indict you or manufacturers. If the EPA really has figured out effective techniques to reduce Goodhart behavior I am very intetested for selfish reasons. Philosophy and Education is my PhD project and I'd like to get into policy. Education, particularly evaluation stuff, is absolutely plagued with Goodhart effects and I am not above stealing from the EPA.

I suppose the fleet scores are improving because the really inefficient stuff is now getting 15 mpg rather than 8 mpg? Ie, the big block pickup trucks? It's really hard for me to believe light duty cars have gotten better. Just pop open a Road and Track from the 90s. Maybe I am cherry picking.

In reply to DaewooOfDeath :

The super small cars haven't seen as much improvement, but the mid size stuff certainly has.  A modern Civic is almost as big as an early 2000s Camry (the Civic is a few inches shorter, but that's about the only size difference).  And the early 2000s Camry doesn't get 40+ mpg on the highway, the modern Civic does. 

DaewooOfDeath said:

In reply to alfadriver (Forum Supporter) :

I suppose the fleet scores are improving because the really inefficient stuff is now getting 15 mpg rather than 8 mpg? Ie, the big block pickup trucks? It's really hard for me to believe light duty cars have gotten better. Just pop open a Road and Track from the 90s. Maybe I am cherry picking.

7mpg improvement may seem minor, but that thought is MASSIVE.  That's almost double the system efficiency.  More like 9mpg vs 8- which is also pretty darned huge.  In terms of heavy trucks, there has been a pretty significant improvement.  Which is why the tiny fuel sippers existed at all.

One other thing- the FE measurements from the 90s are not the same as today.  The 90's scores would go down a little.  That happens as the EPA gets real world feedback and consumers tell them they are generally not getting numbers as advertised.  I know they get a lot of complaints for underperforming, but I doubt they get any compliments when over performance.

I think you're viewing old car fuel economy through rose colored glasses.  Sure, a 5.0 Fox w/2.73s driving a steady 65 might touch 30mpg but the EPA rating for an '88 5.0 mustang is 14/23 and the self-reported average on fuelly.com is 17.3. 

My wife's MINI Cooper weighs about the same as a Fox, only makes slightly less HP (195, I think), and gets a hard-to-believe 29mpg with never-ending short trips around town.  It's just crazy.  A 5.0 would be lucky to see 15 under similar usage (like my 5.0 FFR w/4.11s does).  

spandak said:

Feel free to tell me to leave, but I think some of the examples you are going for are not quite equivalent. The economy cars of past years dont get sold like that anymore. They all have more tech, more safety and more performance. Current economy cars (in the sense of MPG) are lightyears past any of those. Even an old Prius will out MPG anything designed to do the same thing from decades past while being quieter, faster, more comfortable, safer and cleaner.

Trucks are more powerful, larger, safer and more comfortable. I dont know off the top of my head but Ill bet the towing numbers on the ecoboost are significantly more than the older truck. Apart from that, Ill bet the experience is totally different too. Ive towed with old trucks. Technically doing it and comfortably doing it are different experiences.

Funny you should mention a Prius. I owned a beat-to-hell 88 Civic Sedan with a D15 and averaged 41~ mpg. I lived with a 2014 Prius C and averaged  ... 39 mpg. The Civic was a million times better to drive as well. 

That said, I agree with you that the Prius in an accomplishment in the sense that it manages to be only a little worse in fuel economy in spite of all the comfort, safety and technology weighing it down. But this is frankly also partly an externality. 

Cars getting bigger (SUV effect) has created a higher probability of getting hit with something really big, which has sparked a safety arms race, which makes cars even bigger, which requires more safety advances, etc. 

Increased comfort has incentivize greater vehicle use. Safety per mile is better, but the higher number of miles offsets this. Fatality rate, in spite of the huge leaps and bounds in safety technology, was almost exactly flat from 1981 until 2007. Hell, 1964 is only a tiny bit out of whack for normal within that period. It's gotten a little bit better since then, but not nearly as much as you'd expect based on going from two point seatbelts and "what's a crumple zone" to standard side impact airbags and amazing offset crash performances. 

(Externalities are another econ concept, related to the Goodhart Effect but not identical. I'm saying this is a bit of a tangent but another example of how technological advances seem to be chasing their own tails in many senses.)

alfadriver (Forum Supporter) said:

DaewooOfDeath said:

In reply to alfadriver (Forum Supporter) :

I suppose the fleet scores are improving because the really inefficient stuff is now getting 15 mpg rather than 8 mpg? Ie, the big block pickup trucks? It's really hard for me to believe light duty cars have gotten better. Just pop open a Road and Track from the 90s. Maybe I am cherry picking.

7mpg improvement may seem minor, but that thought is MASSIVE.  That's almost double the system efficiency.  More like 9mpg vs 8- which is also pretty darned huge.  In terms of heavy trucks, there has been a pretty significant improvement.  Which is why the tiny fuel sippers existed at all.

I very much appreciate this. Getting 7mpg better in a Civic is not that big a deal. Getting 7 mpg better in a truck is MASSIVE. One reason I think we should replace MPG with gallons/100 miles or similar.