Ford’s Next Infotainment System Could Use Interior Cameras To Read Your Lips And Facial Expressions If It Can’t Hear You Speak

I envy people who can read lips. Being able to see what people are saying, without having to actually hear them, feels like a type of real-life superpower to me. Whenever I’m scrolling on TikTok and come across one of those videos of a lip-reader reciting something said by a celebrity at the Oscars or a professional athlete on the field, I can’t help but watch the whole clip.

Soon, your new Ford might be doing the same thing. According to a patent application filed by the American automaker, it has come up with software that can use interior cameras to read the lips of occupants, in the event normal voice controls aren’t possible (say, like if you’re rolling top-down in a convertible and the wind is interfering with the microphone).

In addition to reading lips, the patent application proposes using the cameras to use facial expressions alone to issue commands. Considering how expressive I get behind the wheel sometimes for no reason at all, I’m not so sure about that one. Either way, it’s worth taking a closer look at how all of this works.

The software side of this patent application is actually pretty straightforward. Ford pitches this specifically for use in a convertible, where, when the top is down, the doors are removed, or the windows are open, the noise in the cabin might increase to the point where cabin microphones might no longer function properly, due to wind buffeting. From the patent:

The vehicle can determine using image data and/or other sensor data that the vehicle is in a convertible state. The vehicle further determines that an ambient noise level in the vehicle is higher than a threshold. The vehicle then switches the mode of operation in order to clearly receive audio input and interpret the audio input. The vehicle may enable a lip reading mode and/or a gesture and facial expression detection mode in order to determine words spoken by a user of the vehicle.

Here’s a flow chart of how the software determines whether there’s enough noise in the cabin to switch to what Ford describes as “enhanced mode:”

In this case, “enhanced mode” represents the mode within the infotainment that uses cameras to start reading lips and facial expressions. In another flow chart, you can see how engineers break down those two functions into two separate modes:

According to the patent application, switching to enhanced mode wouldn’t be totally automatic. In this setup, the driver and/or passenger would be prompted by the infotainment screen to select which modes they’d like to enable—one, both, or neither. Here’s a drawing from the application that shows what that might look like, embedded into Ford’s classic SYNC infotainment layout:

How The Lip Reading And Facial Expression Analysis Works

Ford says it can use cameras to read the lips of the occupants, passing the images into AI that’s been trained on lip reading to convert the lip movements into written words, which the computer can then translate into a command, without ever having actually heard what the person said.

The lip reading mode, if enabled, allows the vehicle to determine what the user is speaking even if the speech is not audible or is partially audible. The one or more cameras of the vehicle can capture the movements of the user’s lips. Other sensors may capture the gesture data and/or facial expressions as the user is speaking. The captured video and sensor data are processed using machine learning algorithms that are trained on large datasets of lip movements and corresponding speech to learn the patterns and nuances of lip reading.

It’s not just cameras that this system might use. The car could also utilize a radar-based sensor to determine lip movements, which sounds pretty neat:

The vehicle may also use acoustic signals to detect facial movements. For example, the vehicle may emit inaudible sound waves and analyze the echoes that bounce back from the user’s lip and mouth. The machine learning algorithms can then interpret the lip movements and translate them into text or spoken words.

The facial expression mode would use the same suite of sensors, but instead of just reading lips, it would track your entire face and head to determine movements and translate those into commands or responses.

As Ford explains it, the facial expression software would be used to determine “whether the user is experiencing difficulty in interacting with the communication system.” From the application:

For example, humans may perform certain gestures such as nodding of the head, shaking of the head, rolling the eyes, etc. in response to verbal communications. In the instance where the user is nodding his/her head, the vehicle may determine that the user is acknowledging a verbal conversation or an audio output via the speakers of the vehicle. The vehicle may then conclude that the current level of audio volume and operating conditions for the communication systems are adequate based on the user gesture and/or facial expressions.

On the other hand, if the user shakes his head or appears confused based on his facial expressions, the vehicle may determine that the user is having an issue with the communication system and may take certain actions such as increasing the audio volume or slowing down the pace of the audio output.

Depending on how you feel about the equivalent of an all-seeing eye constantly watching your every move while you drive, this seems like a pretty neat and useful feature for a convertible like a Mustang or a Bronco with its doors removed. Normally, voice controls can be rendered useless by the wind in these cars, but with a system like this in place, you wouldn’t even have to raise your voice to get a command across.

There are some obvious privacy concerns here—the patent even mentions that the lip-reading and facial detection modes can function off of a cloud-connected server. That means your face and whatever you say might be stored at a data center somewhere and used to train algorithms without you ever knowing. But it’s not like in-cabin cameras and face sensors are a pie-in-the-sky concept—they already exist in some production cars. Subaru’s DriverFocus system, introduced in 2019, for example, uses an infrared camera to monitor driver fatigue and alert them if it detects the driver is distracted or starting to fall asleep.

If anything, this is simply a new idea for how to better utilize those sensors to make life easier for occupants. I’ll give props to Ford for that.

Top graphic images: Ford; DepositPhotos.com; Warner Bros.