Once upon a time, firmware updates on a car were a fraught process, involving expensive proprietary programming tools or literally ripping out and replacing chips. These days, by virtue of wireless connectivity, it’s possible to remotely update cars over the air. It’s convenient for customers and automakers, right up until it goes wrong, as Rivian has found out.
The latest Rivian update was pushed out on Monday for R1T and R1S vehicles, labelled “2023.42.0” It was intended to improve proximity locking and unlocking and fix a variety of other bugs. Customers unlucky enough to be first in line for the update found that the update would stall at 90% completion. From there, the infotainment screen would go completely black and become unusable, with the instrument display affected as well according to some reports.
Affected users can rest assured that their vehicle is not bricked; indeed, it’s still possible to drive the vehicle in this condition. However, there’s no way to control the HVAC system with the infotainment screen dead, and in cases where the main cluster is similarly dead, there’s no speedometer available either. Owners stuck in a bind in this situation would be best advised to buy a thick jacket and use a mobile phone GPS speedometer app. Rivian has advised customers that vehicle resets or sleep cycles will not solve the problem.
Once 2023.42 fails to install (which it will at 90%), you are left with 2 black screens. Vehicle is drivable but with no infotainment. Even though the screens are black, the vehicle is not bricked. pic.twitter.com/lMrCWEIOTf
— RivianTrackr (@RivianSoftware) November 14, 2023
One notable point is that the reversing camera still works under these condtions, popping up on the infotainment screen and operating otherwise as normal. The Rivian mobile app may be of use, too, reportedly retaining functionality for some users with regards to ancillary features.
In a statement posted to Reddit, Rivian put the issue down to human error—” a fat finger where the wrong build with the wrong security certificates was sent out.” These certificates are used to digitally “sign” a firmware update so that the vehicle can verify it is a trusted update from the automaker. It’s a routine security measure to stop random hackers flashing a vehicle with malicious software. However, in this case, it suggests that Rivian’s vehicles weren’t set up to gracefully reject a build with improper certificates, which resulted in the screen failures.
The update campaign was cancelled when the fault was realized, with Rivian planning to restart it in due course with the appropriate build with the proper certificates. Rivian has reported that it has developed a solution and will be notifying impacted owners of the next steps by email. As yet, it’s unclear whether affected vehicles will be able to receive an over-the-air update, or whether they will require an update by physical connection. It’s not uncommon for a bad over-the-air update to cripple an affected system to the point it loses connectivity, at which point it becomes impossible to update remotely.
The Autopian reached out to Rivian for comment regarding the fix, with the automaker confirming that remote update will be used to fix the affected vehicles. “Our customers’ experience matters tremendously to us. We’re making an over-the-air update available for an R1T and R1S infotainment issue that will restore full functionality to the 3% of Rivian vehicles that were impacted,” said Wassym Bensaid, Senior Vice President of Software Development at Rivian. “From here, we’re deeply evaluating our process and quality checks to prevent this from happening again.” The automaker also noted that the issue was identified within 90 minutes and further rollout was halted, with only 3% of customers affected. It credits its vehicle’s compartmentalized design for safety critical systems for allowing its vehicles to still function despite the faulty update.
Overall, it’s a rough lesson for a nascent company. Send out a bad update over the airwaves, and it’s possible to annoy a lot of people really quickly. The same thing wouldn’t happen with a physical roll-out—any dealer tech would stop installing an update that killed a screen immediately and report back to head office. Operating in today’s fast-paced wireless world requires more care to avoid causing headaches at scale.