My Prius was one of 1.9 million vehicles recently recalled in order to fix a software glitch.
When I first heard the news, just weeks after Toyota announced another recall of select seat covers, I was more than a little curious as to what’s going on.
For a long time Toyota has been a well trusted, savvy manufacturer, renowned for its lean practices and Six Sigma quality programs. So what, if anything, has changed?
I called my local Toyota dealer, booked a service appointment the same day, and less than an hour later found myself chatting with Manny, my local service technician.
Manny showed me where the problem was located – a hidden portal under my car’s instrument panel: the USB port. Over the next two hours my Prius’ brain would slowly be replaced with a less buggy one downloaded from a laptop.
While I was waiting, I headed over to priuschat.com, which was lit up in full geek mode. Much talk of inverters, transistors, thermal stress and Boost Converter Voltage – and like any Internet water cooler, a few juicy conspiracy theories as well.
I’ve been in the software development world for nearly three decades, first as a programmer (yes we were programmers, not developers, back then), then as a system administrator, and finally as a marketer of software and application lifecycle management solutions. Those are the solutions that are supposed to prevent recalls like these from happening.
So while I would never presume to know what was *really* happening back at Toyota headquarters, I have a few theories of my own.
The first law of software is that there is no such thing as bug-free software. It’s a variant of the familiar “last mile” problem, only with software it’s the last .001 percent problem. Debugging the first 99.99 percent of software defects is within typical cost constraints for any manufacturer. Then things start to get interesting.
Software has grown so complex—with 100 million lines of code in the average luxury vehicle—with so many variations of real-world test conditions (near infinite variations of temperature, speed, acceleration, torque, etc.), that it is no longer cost-effective to test all software and systems pathways across all operating conditions.
As a result, our thinking about quality management has evolved. Although individual tests may pass or fail, software quality itself cannot be captured with a simplistic pass/fail, on/off indicator. Instead, it has become something that manufacturers must continuously manage against risks, hazards, and cost. It’s a giant probability exercise, managed over thousands of dependent and independent variables.
The documents that Toyota filed with the NHTSA (National Highway Traffic Safety Administration) reveal three important facts:
- The recall stemmed from transistors that, at high operating temperatures, could potentially shut down the car’s hybrid system. In other words, the buggy bits surfaced at the nexus of hardware, software and real-world operating conditions.
- The recall impacted a single model year produced over a four-year period, from 2010 to 2014 . The defect was sneaky and small enough to persist through multiple vehicle validation cycles.
- To date, the defect has caused no injuries or accidents. The recall was both pre-emptive and voluntary.
To me, these facts point to a fully functioning quality management system at Toyota. The defect could have been in that .001 percent category that was too low-risk and costly to identify pre-release. With the additional insight gleaned from millions of vehicles on the road, it was ultimately discovered and elevated to actionable status.
We will probably never know for sure. But the fact that Toyota’s stock price actually rose after the news makes me just a little bit happier to be a Prius owner.