Home » Rivian Bricks Infotainment Systems Thanks To “Fat Finger” Mistake

Rivian Bricks Infotainment Systems Thanks To “Fat Finger” Mistake

Rivian Bad Update Ts2
ADVERTISEMENT

Once upon a time, firmware updates on a car were a fraught process, involving expensive proprietary programming tools or literally ripping out and replacing chips. These days, by virtue of wireless connectivity, it’s possible to remotely update cars over the air. It’s convenient for customers and automakers, right up until it goes wrong, as Rivian has found out.

The latest Rivian update was pushed out on Monday for R1T and R1S vehicles, labelled “2023.42.0” It was intended to improve proximity locking and unlocking and fix a variety of other bugs. Customers unlucky enough to be first in line for the update found that the update would stall at 90% completion. From there, the infotainment screen would go completely black and become unusable, with the instrument display affected as well according to some reports.

Vidframe Min Top
Vidframe Min Bottom

Affected users can rest assured that their vehicle is not bricked; indeed, it’s still possible to drive the vehicle in this condition. However, there’s no way to control the HVAC system with the infotainment screen dead, and in cases where the main cluster is similarly dead, there’s no speedometer available either. Owners stuck in a bind in this situation would be best advised to buy a thick jacket and use a mobile phone GPS speedometer app. Rivian has advised customers that vehicle resets or sleep cycles will not solve the problem.

One notable point is that the reversing camera still works under these condtions, popping up on the infotainment screen and operating otherwise as normal. The Rivian mobile app may be of use, too, reportedly retaining functionality for some users with regards to ancillary features.

ADVERTISEMENT

In a statement posted to Reddit, Rivian put the issue down to human error—” a fat finger where the wrong build with the wrong security certificates was sent out.” These certificates are used to digitally “sign” a firmware update so that the vehicle can verify it is a trusted update from the automaker. It’s a routine security measure to stop random hackers flashing a vehicle with malicious software. However, in this case, it suggests that Rivian’s vehicles weren’t set up to gracefully reject a build with improper certificates, which resulted in the screen failures.

The update campaign was cancelled when the fault was realized, with Rivian planning to restart it in due course with the appropriate build with the proper certificates. Rivian has reported that it has developed a solution and will be notifying impacted owners of the next steps by email. As yet, it’s unclear whether affected vehicles will be able to receive an over-the-air update, or whether they will require an update by physical connection. It’s not uncommon for a bad over-the-air update to cripple an affected system to the point it loses connectivity, at which point it becomes impossible to update remotely.

The Autopian reached out to Rivian for comment regarding the fix, with the automaker confirming that remote update will be used to fix the affected vehicles. “Our customers’ experience matters tremendously to us. We’re making an over-the-air update available for an R1T and R1S infotainment issue that will restore full functionality to the 3% of Rivian vehicles that were impacted,” said Wassym Bensaid, Senior Vice President of Software Development at Rivian. “From here, we’re deeply evaluating our process and quality checks to prevent this from happening again.” The automaker also noted that the issue was identified within 90 minutes and further rollout was halted, with only 3% of customers affected. It credits its vehicle’s compartmentalized design for safety critical systems for allowing its vehicles to still function despite the faulty update.

Overall, it’s a rough lesson for a nascent company. Send out a bad update over the airwaves, and it’s possible to annoy a lot of people really quickly. The same thing wouldn’t happen with a physical roll-out—any dealer tech would stop installing an update that killed a screen immediately and report back to head office. Operating in today’s fast-paced wireless world requires more care to avoid causing headaches at scale.

Share on facebook
Facebook
Share on whatsapp
WhatsApp
Share on twitter
Twitter
Share on linkedin
LinkedIn
Share on reddit
Reddit
Subscribe
Notify of
56 Comments
Inline Feedbacks
View all comments
ScottyB
ScottyB
7 months ago

I feel lucky to have a vehicle with CarPlay yet unable to receive over the air updates so everything that’s working can’t get blown up.

Grippy Caballeros
Grippy Caballeros
7 months ago

Minor peeve but my god I’m tired of resetting my radio presets every time Ford does an update on my Mach E. I finally took a picture of the screen so I’m not trying to remember where my favorite industrial/grindcore channel went.

Oldskool
Oldskool
7 months ago

One major reason I refuse to buy newer cars.

Ok_Im_here
Ok_Im_here
7 months ago

someone trying to get an update out on a Friday to be fresh over the weekend by Monday it would seem.

Elhigh
Elhigh
7 months ago

This just doubles down my commitment to driving stripped models. Bottom-tier cars with just the basics: go, stop, turn and reverse, and maybe air conditioning, have a lot less to go wrong. And with so little added equipment shoved into the vehicle, you can get by with the most basic of controls: knobs. Buttons. Ooh, a slider for those of you who need speeds in between speeds.

Just saying.

DialMforMiata
DialMforMiata
7 months ago

I don’t know that “Oopsie, butterfingers! My bad!” is necessarily the image that a forward-thinking builder of luxury vehicles wants to project when it remotely disables its customers’ expensive vehicles due to a pretty basic mistake.

Dodsworth
Dodsworth
7 months ago

Cars have become cell phones. When they have an update it’s best to wait a few days and see what happens to the other lab rats.

Jho'nuquas
Jho'nuquas
7 months ago

Aaaaand another reason not to buy an electric vehicle

Trust Doesn't Rust
Trust Doesn't Rust
7 months ago
Reply to  Jho'nuquas

ICE cars get over-the-air updates as well, but nice try.

Gerontius Garland
Gerontius Garland
7 months ago

Only new ones. I wouldn’t buy any of those, either.

Geoffrey Reuther
Geoffrey Reuther
7 months ago
Reply to  Jho'nuquas

Shit, Chevy bricked the infotainment unit in my all-gas Sonic more than once with an update.

This is most definitely not an EV specific issue. Programmers derp all the time.

Personally I don’t like OTA updates because if something like this were to happen, I’d want it to happen, say, at the dealership while I am specifically there to get an update, so they can fix their own FUBAR. Not when I’m, say, trying to get home after a rough day at work.

The happy medium is downloadable updates that can be loaded from an SD card which a couple of my cars can do.

Beatle
Beatle
7 months ago

I’m guessing Rivian is the same as Tesla in that updates only download over wifi. In general, that means you’re at home. Still not as ideal as having the dealer fix it or give you a loaner while you’re there, but it shouldn’t happen in the middle of a trip.

Geoffrey Reuther
Geoffrey Reuther
7 months ago
Reply to  Beatle

Chevy did it via 4G, so it could trigger at any time. Once it tried updating when I turned it on while parked 4 levels below ground in a parking garage. Probably the safest place to be when that kind of update is attempted.

MrLM002
MrLM002
7 months ago

Stuff like this is why I prefer not over the air updates.

Thomas Metcalf
Thomas Metcalf
7 months ago
Reply to  MrLM002

Agreed. I really hope that the idea of screens control everything is dead by the time I have to buy a new car.

Geoffrey Reuther
Geoffrey Reuther
7 months ago
Reply to  Thomas Metcalf

At the very damn least, put climate controls back to their own separate controller/buttons. This is the 2nd day in a row I haven’t been able to adjust the climate controls in my Santa Fe because the stereo would only partially boot, and though the climate controls have physically separate buttons, they’re still partially controlled through the head unit. I’ve gotta disconnect my battery for a bit to see if that resets the HU, and if not, it may be time to do some… creative engineering.

Thomas Metcalf
Thomas Metcalf
7 months ago

As a Canadian, I could not deal with non-functional HVAC. It would be downright dangerous to drive with a frosted up windscreen.

Geoffrey Reuther
Geoffrey Reuther
7 months ago
Reply to  Thomas Metcalf

And the HU reset fixed it… until I let it sit overnight. So the new (cheap) fix is to reset the HU when I turn on the car every morning (takes about 15 seconds and requires me to keep a ball point pen in the car). My USB stick will never again be able to remember where it left off, but that’s a small price to pay for not having to completely reengineer what was installed in the car. Maybe I’ll replace the USB stick with one of my backup iPods.

Elhigh
Elhigh
7 months ago
Reply to  MrLM002

Just get the damn thing right and then stop fiddling with it.

Millermatic
Millermatic
7 months ago

Affected users can rest assured that their vehicle is not bricked; indeed, it’s still possible to drive the vehicle in this condition. However, there’s no way to control the HVAC system with the infotainment screen dead

This is one of the _many_ reasons it’s a stupid idea to get rid of physical controls.

Sklooner
Sklooner
7 months ago
Reply to  Millermatic

Sadly a lot of physical controls just route through the computer so you would be no better off, except you could push a button and swear

Geoffrey Reuther
Geoffrey Reuther
7 months ago
Reply to  Sklooner

*nods sadly in Santa Fe*

Jb996
Jb996
7 months ago

Doesn’t the FMVSS require all cars to have speedometers? A NHTSA with guts would immediately issue a recall, and fine Rivian for thousands of violations of vehicle safety standards.

After that, then you would see Rivian and others actually implement software quality control.

Christo Arvanitis
Christo Arvanitis
7 months ago

Happened to my friend’s R1S yesterday….

A. Barth
A. Barth
7 months ago

” a fat finger where the wrong build with the wrong security certificates was sent out.”

Michael: “I always do that. I always mess up some mundane detail.”

Peter: “THIS IS NOT A MUNDANE DETAIL, MICHAEL!”

Geoffrey Reuther
Geoffrey Reuther
7 months ago
Reply to  A. Barth

Lumbergh: “Yeaaaaaahhhhhh….”

Jack Beckman
Jack Beckman
7 months ago

Rivian’s aren’t cheap. Why don’t they have a way to revert to the previous version built in? Especially if they are doing over-the-air updates that could fail just due to a communication error. And as already mentioned here, another reason for physical HVAC controls.

V10omous
V10omous
7 months ago

Why on earth would they not test this on an internal unit before sending it to customers?

Also, what a world when corporations are posting official statements to Reddit.

TOSSABL
TOSSABL
7 months ago
Reply to  V10omous

Funny we just talked about OTAs yesterday. I didn’t even consider this scenario

V10omous
V10omous
7 months ago
Reply to  TOSSABL

Yeah, this honestly wouldn’t be *awful* if the speedometer and climate controls weren’t affected.

We still haven’t seen the true nightmare scenario where the car itself is bricked.

Taco Shackleford
Taco Shackleford
7 months ago
Reply to  TOSSABL

Scenarios like this are exactly why I don’t want an OTA car. This is one of the lower risk examples, but if mistakes made by some buffonn months after I’ve departed the dealer keep me from driving, I may end up getting it towed back to the dealer and leaving it there.

Cyko9
Cyko9
7 months ago
Reply to  V10omous

With Twitter on the wane, everyone is wondering how to communicate with the masses easily. For the price of entry, Rivan should have a user messaging service they could reach out with.

Drew
Drew
7 months ago
Reply to  Cyko9

They could send a message to show on the infotainment screens of affected veh…oh, wait.

Cheap Bastard
Cheap Bastard
7 months ago
Reply to  Cyko9

What’s wrong with text messages?

10001010
10001010
7 months ago
Reply to  V10omous

Apparently Rivian is using the same QA processes as Microsoft before shipping updates #fullsend

Duke of Kent
Duke of Kent
7 months ago
Reply to  V10omous

I think that the statement on Reddit was the most surprising part of the story for me. Doesn’t every company in the world have their own website? Why not post it there? Are they trying to be cool and hip like Tesla where the CEO tweets Xes every thought that enters his mind?

Ben
Ben
7 months ago
Reply to  Duke of Kent

Maybe someone also fat-fingered a website update and crashed it.

Andrew Wyman
Andrew Wyman
7 months ago
Reply to  V10omous

First thing I thought out. At least test this on a few internal center stacks before sending out. Don’t sacrifice reliability for speed.

Thomas Metcalf
Thomas Metcalf
7 months ago
Reply to  V10omous

The world has gone stupid. If you’ll excuse me, I am going to tie an onion on my belt and go yell at a cloud.

TOSSABL
TOSSABL
7 months ago
Reply to  Thomas Metcalf

Don’t forget to string a potato and wear it around your neck!
#youllbefine #donttakechances

Jb996
Jb996
7 months ago
Reply to  V10omous

Why? Why should they?
Right now software on all EVs seem to allow the programmer to directly issue an update, and if something goes wrong, the company issues a half-hearted Reddit “My Bad”.

Until either would-be customers (as in, haven’t paid yet) or government fines force them to, why should they take on the cost of rigorous software quality control?

Last edited 7 months ago by Jb996
Óscar Morales Vivó
Óscar Morales Vivó
7 months ago

Hasn’t this happened to Tesla a bunch of times already?

Rusty S Trusty
Rusty S Trusty
7 months ago

You know what never went black after software updates or, for that matter, never needed software updates in the first place? Actual physical buttons and knobs. What was ever wrong with those?

StillNotATony
StillNotATony
7 months ago
Reply to  Rusty S Trusty

You know what else never went black after software updates or, for that matter, never needed software updates in the first place?

Wheelbarrow of shrimp.

And that’s why the Autopian is the leader in automotive journalism.

Goof
Goof
7 months ago
Reply to  StillNotATony

There’s still not a membership level which guarantees an actual wheelbarrow of shrimp. Not even a toolbox of cocktail sauce! I have to personally hold off on such accolades until such membership perks are offered.

Angrycat Meowmeow
Angrycat Meowmeow
7 months ago
Reply to  Rusty S Trusty

I get where you’re coming from, but in this decade it’s absolutely absurd to expect to buy a $90k electric vehicle and be presented with buttons and sliders straight out of 1992. We can whine and express our displeasure all we want, but the features and creature comforts we expect out of our cars today just won’t play with old school stuff. Even if it had redundant controls, it wouldn’t surprise me if they were routed through the infotainment and would become inoperable in this scenario anyway.

Rusty S Trusty
Rusty S Trusty
7 months ago

Having to choose between risking taking your eyes off the road to look at a touchscreen or finding a place to pull over just to change the temperature in your $90,000 vehicle doesn’t seem like much of an advancement to me

Angrycat Meowmeow
Angrycat Meowmeow
7 months ago
Reply to  Rusty S Trusty

That argument is tired and doesn’t really hold water anymore. My car has buttons, but to operate the majority of them you need to look. To change the drive mode, turn on/off parking aids, auto stop, seat heating/ventilation etc you have to look at a row of featureless, identical buttons and figure out which picture means what. So many things are automated now, who cares if they’re buried in a touchscreen? Why are you messing with the headlights when they’re automatic? The amount of cars I see driving sans headlights at night tells me many drivers are turning knobs they shouldn’t be. Leave it in auto. Media controls, and usually lots of other stuff, are available on the steering wheel.

I’m not firmly in one camp or another, I’m just over this “old man yells at cloud” stance on screens. Nobody is getting their 1991 Accord dashboard back and it’s for reasons both good and bad. In the end as long as the UI is well designed it doesn’t make a difference whether the button is on a screen or clumped together with a bunch of similar buttons at the bottom of the dashboard.

Geoffrey Reuther
Geoffrey Reuther
7 months ago

Oh it very much matters. When you have actual physical buttons, your fingers can tell where one ends and the next begins. After a couple weeks of driving any given car, you should not need to look at your climate controls if you have physical buttons; muscle memory will take over and you can do it while keeping your eyes on the road. But if those controls are on a touchscreen, because there is no physical difference along the entire screen’s surface and the car can’t figure out if you’re just groping for something or you really do want to fire your photon torpedoes at the Klingons, you do actually need to look at what you’re doing.

SooperDooperPooperScooter
SooperDooperPooperScooter
7 months ago

2019 GTI owner here:
Drive mode – button next to shifter
Parking aids – steering wheel button
Auto stop – button next to shifter
Seat heater – center stack button

Still a touchscreen in the center stack for radio related items. This is the major reason I bought a Mk7.5 over a Mk8 earlier this year.

Baking every function into a touchscreen is neither necessary nor effective. Screens die, lack physical gaps between buttons, get greasy/washed out in the sun, and are generally a pain in the ass to muddle through while driving. I can find all the above functions while still trying to figure out what that vanity plate ahead of me in traffic is supposed to mean.

Last edited 7 months ago by SooperDooperPooperScooter
Cheap Bastard
Cheap Bastard
7 months ago

“in this decade it’s absolutely absurd to expect to buy a $90k electric vehicle and be presented with buttons and sliders straight out of 1992.”

What’s absurd is I’m still rolling on rubber like its 1892, where’s my flying car? And my Mr Fusion to power it?

Sebastian Bear
Sebastian Bear
7 months ago

Huge infotainment screens that control very basic functionality aren’t what I’d consider “creature comforts”. They are harder to use, more distracting, and fail way more easily and in much more dramatic fashion than physical buttons and knobs do. They solve a problem that nobody had. “Oh gee, I sure wish it was more complicated to adjust the temperature in my car”. Buttons and knobs work. Big screens in lieu of buttons are purely a cost-cutting measure.

Angrycat Meowmeow
Angrycat Meowmeow
7 months ago
Reply to  Sebastian Bear

 fail way more easily and in much more dramatic fashion than physical buttons and knobs do.

The same can be said of almost any technological advancement. Dynamic chassis control? VVT? Forced induction? Fuel injection? Electronically controlled autos? It all sounds like my dad saying “You can’t work on these dang new computer cars! Back in the 70’s I could rebuild a carb with my eyes closed!”. Everyone just wants to be a stick in the mud and pine for the olden days instead of learning or adapting. The “SCREEN BAD” rhetoric is like watching a bad infomercial. I picture a bunch of people randomly jabbing at a screen like they’ve never seen one before while pulling their hair out.

It’s not that bad, and if you’re tech-averse, nobody is telling you that you can’t keep driving your 90’s Whatever SE (at least in the US).

I mean, look at this thing. There is no argument that a screen is better suited to controlling the functions of any modern car than 75 knobs and buttons all around the dash and console.

https://en.wheelsage.org/pontiac/bonneville/1987-91/gallery/y08kb3

Sebastian Bear
Sebastian Bear
7 months ago

The difference between forced induction, VVT, etc. and screens to control basic functions is that they solved problems that existed. They improve efficiency, power, pretty much everything besides reliability. Screens have their place in the automotive world, just not to control climate, headlights, windshield wipers, etc. Buttons do everything that screens do in this context, except they do them better. What benefit does a screen have to buttons in the context of climate control?

Also, the Pontiac is not and would not be the norm. That setup is way too complex, hard to use, confusing, and just isn’t as effective as a more reasonable number of buttons would be. Sort of like a big screen to control your climate. Basic functions should be controllable via easy to use, tactile buttons. The rest of it can be hidden in menus.

Last edited 7 months ago by Sebastian Bear
Mike Harrell
Mike Harrell
7 months ago

“…nobody is telling you that you can’t keep driving your 90’s Whatever…”

Well, I certainly can’t. I’ve never owned a car that new.

Ben
Ben
7 months ago

See, the thing is I’m not tech averse. I like having a touchscreen for things like tweaking the way my heated seats work and telling me who is calling when my phone rings (which, I will note, I answer using a physical button on my steering wheel so I don’t have to look at the screen any longer than it takes to read a name).

I do not like having to use a touchscreen for things I need to interact with while I’m driving. Touchscreens are an objectively bad interface for that and it needs to stop.

As usual, taking either design philosophy to its most extreme form is bad. 75 buttons is a bad user interface. That same functionality entirely on a touchscreen is also a bad user interface. There’s a right way and a wrong way to integrate a screen in a car and far too many manufacturers are doing the latter.

56
0
Would love your thoughts, please comment.x
()
x