A History Of GM Software, And How A New ‘Quality Lab’ Aims To Fix The Company’s Recent Problems

The term software-defined vehicle (SDV) might be a relatively new one, but putting software into vehicles to control a wide range of systems actually goes back nearly 50 years. I personally spent more than 17 years in automotive product development through the 1990s and early 2000s mostly working on software. However, over the past decade, the nature of the software and electronic architecture of vehicles has changed dramatically and that has necessitated totally different ways of working. Making such fundamental changes in a large organization isn’t easy, and legacy automakers like General Motors, Volkswagen, Ford and Volvo have had major struggles in recent years. Some of GM’s software challenges have been particularly public and led to the company establishing a new software quality lab to try to address those problems.

The ‘Good’ Old Days

Up until the 2012 production launch of the Tesla Model S, most of the auto industry handled software in pretty much the same way. The code was deeply embedded and closely tied to the hardware of the vehicle. Embedding electronic control systems into vehicles began in the 1970s when processors and memory were still quite expensive and didn’t offer much in the way of performance. Some of the earliest electronics handled systems like ignition and fuel injection as automakers struggled to meet early emissions and fuel economy regulations. 

The cost and limited performance capabilities of those early chips meant that code had to be extremely efficient to do as much as possible with as few bits as possible. Rewritable flash RAM was still a couple of decades in the future so the code was written into read-only-memory (ROM) when the chip was manufactured and could never be changed without swapping out the entire electronic control unit (ECU). Over-the-air (OTA) updates were also a fantasy that was solely a joke as recently as the late-1990s. 

The electrical/electronic (E/E) architecture of the vehicle, which consisted of the wiring harness and ECUs, was fundamentally similar for everyone and remained so until the last decade. The only thing that changed about the E/E architecture was size and scope. The wiring harness was always there to distribute power to accessories like lights and to bring signals from switches like the brake pedal. 

At first we had just an engine ECU, but over the next several decades, each time a new feature was developed such as anti-lock brakes or automatic climate control the E/E architecture grew. Automakers would buy a feature from a supplier like Bosch, Continental, Magna, ZF or TRW and it would include some sensors and actuators and an ECU with embedded software. 

Then the automaker would have to figure where to place these pieces in the vehicle and tie them into the wiring harness. Before the term infotainment even existed, in-vehicle media and navigation was pretty much self-contained. By the mid-2010s, some vehicles had upwards of 120 ECUs and four miles or more of wiring. 

While basic connectivity like controller area networks (CAN) and FlexRay was added that enabled some data sharing, the reality is these systems were largely siloed. By 2000, some of these ECUs contained some flash RAM that could be updated (“flashed”), the updates could still only occur with a physical connection to the vehicle from a special tool, usually at a dealer. These systems were generally only updated if a major bug, safety issue or regulatory compliance problem had to be addressed. 

When a new vehicle program was kicked off, so too was the software development process at automakers and suppliers. A date, typically about six weeks ahead of start of production, was the drop dead date to finish validating the software and deliver it to the factory. Beyond that date, any additional changes usually had to wait for a new model year, mid-cycle refresh or next-generation. Aside from recalls, vehicles in the field did not ever get updates and new features were something that owners added from the aftermarket. The software running when the car went to the scrapyard was typically the same it had when it rolled out of the factory. 

[Ed Note: Back when I worked at Chrysler, I recall there being a software problem on some car (I think maybe the Dodge Dart?), and the result was the dreaded “Yard-Hold.”  Someone with the username “garyfirestorm” has a nice little blurb about yard-holds on ycombinator:

The official term is yard hold. This is nightmare for any engineer who’s part is responsible for yard hold. The vehicles are made, but not allowed to be shipped to a dealership. And everyone’s breathing down your neck unless the issue is fixed and they are allowed to leave. Issues can be anything from simple part squeaking to major design flaw. Each vehicle is test driven over a course before the manufacturing teams give it a ‘ship it’ tag. Door fit is a major issue for anyone and everyone. You simply cannot engineer doors to fit right, the assembly line guys always have to pull it, twist it and ‘align’ it. Tolerances exist, but the tighter you go the more parts you’ll reject, the more the cost of that part will be i.e. cost of your car will go up. Everything can be fixed and it’s never too late, the only problem is the cost of the fix. The later in production a fix is issued, the higher is the cost.

I remember Chrysler asking for volunteers to go to the yard and manually flash hundreds of cars in the yard-hold. Its’ amazing how today the problem likely could have been solved in a matter of moments instead of weeks, and heck, maybe the cars could have been shipped with the OTA software update soon to be sent out. -DT] 

Then Came The Model S

When Tesla developed the Model S, it had no legacy architecture to iterate on. Its previous car, the Roadsterm was an electric derivative of the Lotus Elise architecture and even though much of the car was bespoke, it wasn’t designed for mass production. Elon Musk is fond of the idea of working from first principlesm and that is often, albeit not always, a good place to start when you have nothing before to evolve.

The Model S had far fewer ECUs than typical vehicles of the time — believed to be a couple of dozen — and they were more tightly networked. The software architecture was also explicitly designed for easy remote updating capability. Up to that point, other vehicles could generally only do OTA updates to their telematics control units. Everything else in the vehicle had to be updated through a physical connection and those updates were typically executed by dealer service technicians. 

Having technicians reflash the software is a huge expense for automakers and they generally try to avoid it except as a last resort. On the other hand, it’s also a major profit center for dealer service departments so they are in no rush to see a change. 

Interestingly, GM was actually one of the first automakers to execute OTA updates. At least as far back as 2009, the automaker had pushed updates to telematics and infotainment software to vehicles in the field through its OnStar platform. In 2010, GM even experimented with updating other vehicle systems on the Chevrolet Volt captured test fleet (CTF).

At GM, the CTF consists of several hundred early production units that are full production specification and driven by employees for several months before customer deliveries start (Ed Note: At Chrysler this is called the “Fast Feedback” fleet. -DT]. Following this test, GM ultimately made the decision not to implement the expanded OTA updates in production due to lack of confidence that it could be done reliably. 

Tesla pushed out its first OTA update for the Model S in September 2012, just about two months after deliveries began. That first update addressed some infotainment issues but also included changes for the range estimation. In the years since, Tesla has maintained a steady cadence of updates for everything from performance changes to driver assistance systems and holiday light shows. 

Vehicle Intelligence Added

In 2019, GM launched its “vehicle intelligence platform” (VIP), also known internally as the Global-B E/E architecture. For the first time, VIP was explicitly designed to enable OTA updates of most vehicle systems. It included a gigabit ethernet network and larger domain controllers that replaced multiple discrete ECUs. While this wasn’t a fully modern architecture with centralized compute or zone controllers, it did see some reduction in the total number of ECUs compared to the previous generation. Among the other major changes was significantly increased data bandwidth to deal with the increasing number of sensors around the vehicles and a design that factored in cybersecurity concerns. 

The wait for  VIP was one of the reasons that GM didn’t expand availability of its Super Cruise hands-free driver assist system after first launching it on the Cadillac CT6 in 2017. The 2021 Cadillac Escalade which used VIP was the second vehicle to get Super Cruise and it has since been expanded to many other models. The old architecture didn’t have the performance or data bandwidth to support most of the updates done to Super Cruise in the last several years. The only other models that eventually got the original system with the old E/E architecture are the Cadillac XT6 and Chevrolet Bolt EUV and those are still limited to about 130,000 miles of divided highways and no auto lane changes. 

That VIP architecture is still being used on new GM vehicles launching in 2025 although it has continued to evolve, especially with the launch of the Ultium-based EVs. While the software platform for VIP was updated from prior generations, the vehicles that have been launched so far still don’t use the latest and greatest generation software architecture. 

What About Ultifi?

Back in 2021, GM announced Ultifi, it’s next generation software platform. Ultifi was meant to enable a separation between the hardware and software layers of the vehicle systems. The Ultifi branding has since been abandoned but the underlying platform is still being developed and the first iterations should arrive in new vehicles soon.

Back in my engineering days, software was embedded on the hardware. Code would directly read from the physical ports to get sensor signals and then write to other ports to command the actuators. Changing the hardware would often require major software rewrites and revalidation. 

Ultifi and other similar platforms were meant to be more like modern operating systems such as Google’s Android or Apple’s iOS. It would utilize a middleware layer that provided a suite of application programming interfaces (APIs) for the hardware. When using an Android phone (the same is also true for Apple’s iOS on the iPhone), most applications can be run unchanged regardless of whether the phone is running a Qualcomm Snapdragon 8 or Mediatek Dimensity 9000 or Google Tensor 3 processor – similarly, the same code can run on multiple generations of Apple’s A-series chips.

The top level application layer would call an API to camera, radar, wheel speed, powertrain torque or other signals. When it wanted to command the brakes, steering or powertrain output, the applications would call other APIs. The low-level software would be more coupled to the hardware and feed data to and from those APIs. In this way, the hardware and software don’t need to stay in sync. An automaker could change to a different processor or different sensor and not have to update applications. 

Only that low-level layer needs to change.The same is true for updating applications, within the limits of the hardware capability, the applications can be updated independently. These platforms also support running virtual machines with entirely different operating systems. For example there might be a real-time operating system like QNX or Wind River running in one VM for the instrument cluster while a separate VM for the infotainment could be running Android Automotive. 

This separation provides a number of potential benefits. Both the hardware and software can now be developed at a faster pace and updates can be deployed independently. It also makes it easier for  third parties which might be traditional suppliers or new types of service providers to create applications to run on the platform. The addition of new services and applications is a key to automakers hopes of creating new revenue streams from SDVs. 

Managing it all

However, managing these more complex software architectures isn’t easy. Automakers like GM, Volkswagen, Ford, Volvo and others have spoken at length in recent years about bringing this software platform development in-house. However, all of them have struggled to implement the new organizations required to execute on these plans. While software engineers have been a key part of automotive product development for decades, the organizations and processes for modern architectures to work are fundamentally different. 

VW’s Cariad software division has gone through multiple reorganizations and changes of management over the past three years and programs that were depending on this new software have been delayed by three years or more. The situation has gotten so bad within the VW Group, that the German automaker established a joint-venture with Rivian to use the E/E architecture and software platform from the American upstart for its next-generation vehicles. Volvo delayed the launch of the EX90 and Polestar 3 by about 9 months due to software issues. While Ford hasn’t acknowledged any software issues publicly, insiders have acknowledged that the development of an all-new platform hasn’t gone entirely smoothly and it has likely been a factor in Ford’s delay of a number of new EV programs by 1-2 years and cancellation of others. 

At GM, the software issues have been rampant ever since the late 2021 launch of the GMC Hummer EV. There were numerous reports of software issues on that vehicle, many of which were experienced by the TFL Truck Youtube channel during their brief stint owning one. When the Cadillac Lyriq debuted in mid-2022, it was also rife with buggy software. 

The big disaster though came in late 2023 when Chevrolet debuted the Blazer EV. Just as it was going on sale, media reports from InsideEVs and Edmunds caused GM to quickly issue a stop sale on the Blazer. Edmunds had nearly two dozen fault codes on the Blazer it purchased and an InsideEVs writer ended up abandoning the vehicle he was testing during a holiday road trip when it lost power and wouldn’t charge. It would be another three months before GM had validated fixes to the software and allowed customer deliveries to resume. 

Recognizing The Quality Problem

Even before the Blazer EV software issues became public, GM was clearly recognizing that it had a software quality problem that needed to be corrected. If it couldn’t get its software quality sorted out, it was absolutely going to fail in its efforts to deploy SDVs and EVs in the coming years. Without reliable and robust code, they would have no competitive products to sell and little opportunity to grow revenues. 

In October 2023, GM established a new software quality lab in the Sloan Engineering building at its Warren, Mich. Technical Center campus. This new lab was meant to be a key tool in resetting the company’s software course. I was invited to Warren to tour the new lab and learn about what GM was doing differently.

Bench testing of software has always been a key component of the development and validation process. In the early-1990s we utilized both open and closed loop simulation purely in software and with hardware in the loop. Open loop testing involved feeding in a defined data set and monitoring the outputs. Closed loop testing involves feeding back output data into the inputs. 

For example in open-loop tests of ABS, we would provide a data set of wheel speeds and brake switch and monitor which solenoids activated and when. In a closed loop, modeled the vehicle behavior, feeding the control algorithm outputs into a hardware dynamic model of the brakes and suspension to generate wheel speeds. Those generated wheel speeds would then go back into the next loop of the algorithm. With hardware in the loop, we would have actual brakes and feed the pressure into the dynamic model to get closer to the actual result. 

The New Basement Test Lab

The new GM lab is starting out with closed loop, hardware in the loop (HIL) testing. The HIL system is some of the most elaborate that GM has ever done. Each test bench consists of the entire infotainment system and E/E architecture for a particular vehicle model, all of the displays and controls, and all of the connectivity including wifi, bluetooth and cellular data. The benches can be used in one of three ways. An engineer or technician can sit at the bench and physically manipulate the controls just as if they were in the vehicle. All of the relevant ECUs that would be found on that vehicle including the telematics control unit (TCU), visual control unit (VCU), GPS, advanced driver assist system (ADAS) and more are present on the bench. 

With GM continuing to work on a hybrid post-Covid schedule and many developers working remotely or from other GM facilities around the world, there is also a remote login-in option. Each of the benches is networked and a developer can do anything they can do in the vehicle while sitting anywhere in the world. When a developer needs to test something, they can log into one of the benches, load their latest software build and go to work. Along with the physical controls that are present, there are banks of relays that replicate what a button press does in the car. When an engineer sends a signal to press a button on the dashboard from a remote location, a corresponding relay is energized in place of the physical action. The test benches are also equipped with cameras so that the engineers can see what is happening on each screen in real time. 

Finally, there is an automated testing system. Because the test rigs are networked and connected to GM’s data center, whenever they are idle for a period of time, an automated testing sequence can be launched. The cameras record everything as sequences of thousands of test scenarios are being run on the latest software builds. 

Machine vision systems monitor the behavior of the system during testing and highlight anomalies for engineers to go back and check as well as evaluating timing and other parameters. Generative AI is being used to create new testing scenarios to try to ensure more complete coverage of everything that might happen in the vehicle. On one wall of the lab are a series of flat screens displaying dashboards of how many anomalies are being found and fixed. Red on the dashboards is actually considered a good thing because it means bugs are being found before cars go into the field. 

One of the methods that GM uses to determine what needs to be tested is a quality recorder in vehicles. Currently the quality recorder is only used in GM owned vehicles such as its captured test fleet (CTF). GM employees drive pre-production vehicles in the CTF for several months before they go on sale and if an issue crops up they can insert a USB stick and download all of the data logs. The next stage is to provide a button to users to record a voice message with information about what happened. All of this is then shared with the engineers that work on fixing the issues. Right now, there aren’t any plans to enable the quality recorder in customer vehicles, but nothing is ruled out for the future.

GM has long utilized test benches for evaluating infotainment and instrument cluster systems, but these previous generation systems lacked the remote and automated testing capabilities found in the new lab. An engineer had to actually be present to run a test. Another feature that was lacking in the previous test environment was the ability to do OTA updates. 

The new GM quality lab is equipped with Verizon and AT&T micro-cells and each of the benches includes a 4G or 5G data modem and the antennas that are needed. OTA updates can be staged and pushed out to selected benches to verify that all software and ECUs can be updated reliably. The shark fins on each bench also include the antennas for satellite radio and GPS and those signals can also be broadcast in the lab to test those functions. For example simulated GPS signals can be broadcast to verify that the test system can properly calculate the vehicle’s position and track it as it “drives” in the lab. 

There are more than 300 of these benches now for every available configuration GM sells globally from the Chevy Trax running a QNX infotainment system to the Cadillac Celestiq with its 55-inch cross vehicle display and Android Automotive system as well as systems sold in China. Speaking of Android, GM has also deployed several of these benches in Google’s test labs in Mountain View so they can test their updates before they get pushed to GM. 

The next stage of the evolution of the Sloan lab is virtualized ECU testing. Working with Synopsys, a California based company that specializes in simulation of new chip designs, this will enable GM to test new versions of software on next-generation chips and ECUs before physical prototypes are even available. 

The work going on in the Sloan lab is what GM refers to as sub-system testing. Before code gets to the Sloan lab, it’s tested at the component level on individual ECUs at developer’s desks. Once it passes out of Sloan, it goes to a lab in the basement of the Estes engineering building for full system integration. 

The Estes lab contains racks that are effectively an entire vehicle with the complete E/E architecture and everything electrical or electronic including lights, power steering actuators and everything else. In the aviation industry they refer to a setup like this as an “iron bird,” effectively a complete aircraft that can’t fly. This can be used to ensure everything works, again before it goes into a vehicle. Since the issues with the Blazer, other GM launches have generally gone much more smoothly although still not flawlessly. 

What Comes Next?

Current GM vehicles are still using an evolution of the VIP platform, but GM is moving toward a more centralized compute architecture and also making changes in what types of processors it is using. At the Nvidia GTC conference in March, GM announced that it would be adopting Nvidia’s chips for next-generation ADAS and automated driving systems in place of the Qualcomm Snapdragon Ride chips it is using on current VIP products. 

GM isn’t saying exactly when it will start launching vehicles with Nvidia or even which chips it will use. It will probably be at least another year or two before we see these systems and they will probably use either the Orin or Thor processors to run more sophisticated systems that go beyond what is currently possible with Super Cruise. These may be among the first systems that take advantage of the virtualized ECU testing capabilities. 

As for the overall architecture, GM won’t talk about specifics yet, but it has indicated that it is moving to a more centralized compute architecture in the coming years. Interestingly, that doesn’t include use of the type of zonal architecture used by Rivian or some other manufacturers. In a zonal architecture, there are typically 2-3 central compute units to handle areas like ADAS, infotainment and body control as well as 3-4 zone controllers near the corners of the vehicle The zone controllers handle power distribution to sensors and actuators and functions like signal processing before it goes to the central compute. 

What GM plans to move toward is the centralized compute combined with small ECUs on some of the actuators to handle functions that require minimal latency. For example, traditional ABS or stability control software runs entirely in an ECU attached to the hydraulic control unit. In new generation systems, the ABS control software that determines how braking torque should be managed would run in the central compute while the low level algorithms that directly manage the solenoids in the hydraulic controllers would be located right at the actuator. This is analogous to the human nervous system where most functions run in the brain, but reflexes happen directly at the edge of the nervous system 

Only time will tell if GM’s new testing tools and organizational and process changes will enable it to succeed in the new world of SDVs or even catch up to the pace of Chinese automakers, but what we’ve seen so far certainly seems like a step in the right direction.