When I was a geeky kid, reading all sorts of science fiction, the idea of advanced artificial intelligences came up often. Sometimes in the form of robots, sometimes in the form of a computer without a robotic body, but always something thought provoking as these synthetic brains breached the boundaries of sentience and started to grapple with the mysteries of identity and what it means to be something aware of one’s self. They asked what this thing called “love” is or wished to experience laughter or something equally profound.
What they almost never did was make buyback deals on used BMWs that would later be reneged upon.
And yet, here in our stupid reality, that’s what AI is actually doing, when it’s not too busy giving you bad advice or putting pictures of famous people you find online in skimpy clothes. This particular situation, CBC writes, happened at the BMW Toronto dealership, when 2021 BMW X3 XDrive 30i owner Zack Giacomelli, displeased from having to do expensive repairs on the car, decided to reach out to BMW Toronto to see if they were interested in buying the car back.
Happily, they were! At least, according to BMW Toronto’s online representative, named Quinn.
Quinn offered $27,162.79 for the BMW, covering the remaining balance on the car, with “no money owing, and no cash back,” according to texts with Quinn that Giacomelli shared with CBC. Giacomelli was very happy with this offer, and for the hell of it made a counteroffer of $28,000, which Quinn allegedly stated “sounds reasonable.” Quinn even allegedly suggested that Giacomelli come by the dealership to “lock in today at 3:30.”
Everyone should be happy, right? A deal is about to be struck!
Well, not quite. According to CBC,
“Moments later, Giacomelli said, a BMW Toronto sales consultant called to revoke the offer, explaining that Quinn wasn’t a real person, but rather an artificial intelligence chatbot that had made the offer in error.”
Why did the dealership rescind the offer? Because Quinn is not a human being: Quinn is an AI chatbot.
This of course brings up that perennial philosophical question: so what?
Yes, Quinn is an AI, but it’s also acting on behalf of the dealership, and customers who interact with Quinn have no way of knowing if Quinn is man or machine; just ask Alan Turing about that.
Giacomelli brings up an excellent point about all of this, telling CBC:
“If they’re going to be replacing their employees’ jobs with AI, then they need to be honouring what that AI says.”
Mr. Giacomelli, I couldn’t agree more.
You can’t have it both ways; a business can’t automate a job that was formerly a person with authority to make deals like this and then decide that no, what this machine says while operating as an agent of the dealership cannot be taken as actual, binding interactions with the dealership, like what the human agents they replaced did. How the hell is a customer supposed to know that? And if the AI will say things that aren’t actually what the dealership intends to do, then what’s the point of the damn AI?
If a human employee had made a deal like this, the dealership could likely have to honor it, or at the very least consider it. Just as a human can be an idiot and make bad decisions, an AI can be an idiot as well, and perhaps even could execute idiocy with efficiencies and speeds no human could hope to match.
An AI chatbot may be able to interact convincingly with a customer, but it fundamentally does not understand what it’s doing. It has no idea what cars actually are or how much money is actually worth or, well, anything about life in the world, really. It’s a bunch of branching conditionals and massive databases that delivers the illusion of sentience. The reality is it doesn’t “know” jack feces.
Eventually, the dealership did honor the $27,162.79 offer, so they did end up doing the right thing in the end. But even if they hadn’t, the law seems to be on Giacomelli’s side. Canadian law has already encountered this issue, as a case in 2024 ruled that companies (in this specific case, Air Canada) are indeed responsible for what their AI agents tell customers.
The legal decision makes its point and reasoning quite clear:
“Air Canada argues it cannot be held liable for information provided by one of its agents, servants, or representatives – including a chatbot. It does not explain why it believes that is the case. In effect, Air Canada suggests the chatbot is a separate legal entity that is responsible for its own actions. This is a remarkable submission. While a chatbot has an interactive component, it is still just a part of Air Canada’s website. It should be obvious to Air Canada that it is responsible for all the information on its website. It makes no difference whether the information comes from a static page or a chatbot.”
Where that 2024 case made clear that it “makes no difference whether the information comes from a static page or a chatbot,” it would also stand to reason, and perhaps even more strongly, that it makes no difference if an official communication from a car dealership comes from a human agent or a silicon one.
Hopefully, this case will be noticed by other businesses using a chatbot to interact with clients and customers. You can’t embrace the economics of getting rid of employees in favor of AI without accepting whatever some AI may say or do while representing the organization. AIs are, bluntly, idiots. They can’t help it; they’re not alive, and have no genuine understanding of the world. Is that really what a company wants representing it?
(top image: Jason Torchinsky, YouTube, CBC)
He’s an enigma wrapped in a riddle. An international man of mystery. Hell, we went to the same kindergarten, but no one really knows Jack Feces.
“AI can be an idiot as well, and perhaps even could execute idiocy with efficiencies and speeds no human could hope to match.”
Hold my beer.
I use AI often and now since AI is the first result on a google search, I have been asking gemini stuff I know is true it gets stuff wrong or skips stuff. At this point AI is like a lazy teenager.
I’ve seen it straight up hallucinate sports scores, for example, when I do a google search for something like “March Madness Bracket”. Gemini once told me that the championship game was already played, the final score, and that it was played a week in the future.
As Torch says, it doesn’t “know” anything. It is a computer program essentially following an infinitely complex set of if/then statements and returning the result, and only verifying the result against real data if you prompt it to.
Only if said lazy teenager missed the first half of the conversation and just now came into the room. There are fundamental facts an AI just doesn’t…grok (whips off sunglasses)
Ultimate Negotiating Machine
As someone who is related to the industry (AI, not car dealers), there are ways of letting the human know when they are interacting with AI.
Another question is whether companies want to.
Oh, there’s a way I can always tell for verbal communication.
Start giving it information such as a name, phone number or email address – but pause, as one does naturally when one is expecting another human or write to type new information.
The Chatbot thinks you’re done and interrupts you to confirm when no human ever would.
As far as texting – I never assume there’s an actiual human on the other end anymore unless there’s major lag – which tells me there’s someone out there handling 2-3 people at a time.
I actually meant a proper disclaimer “you are interacting with an artifical intelligence”. Those exist and can be inserted in chats or voice conversations.
But of course, it is optional.
It’s also easy to tell for written communication with business emails.
More and more frequently, people are throwing a half-thought response into AI and copy/pasting what it says is an “improved” response and hitting send. The result sounds and is formatted exactly like you expect from an AI output and absolutely nothing like what you know the person talks like.
I have always had a very personable “voice” in my emails, and combined with the fact that I actually know how to communicate clearly and concisely, it’s becoming more and more valuable.
Worth noting is that the dealer was not going to honour the deal, but changed their minds rapidly after contacted by the CBC.
So basically, they got scared when the media got involved. Which is cheaper than lawyers getting involved, but still.
Yet this is nothing new.
Dealers and other companies have been playing these games with their human reps for decades.
Say whatever it takes to get ’em in the door, then tell them the car they wanted is sold, or the deal they expected is expired, etc – then get them to buy something more expensive.
You only hear about it when someone calls BS, then calls the media (or posts online, these days).
Makes me want to see what kind of trade in offer I can get on my ‘04 Honda Element with 259k on it…
After we AI all of the things, later, a revolution where interacting with other humans, face to face, is going to be “the new way”.
More places will be offering the “luxury concierge experience” of a person soon enough. It will be the perfect excuse to make everything cost more but the person will be cheaper than the chatbot.
“Hopefully, this case will be noticed by other businesses using a chatbot to interact with clients and customers.”
Hopefully, this case will be noticed by other
businessescustomers using a chatbot to interact withclients and customersstealerships and more people get better deals on their trade-ins, thus screwing the stealerships in return for the screwing of customers they’ve been doing for decades.Hilarious.
I take a special glee in every company that gets screwed over by their own AI implementation.
I love the ones where they have their entire database deleted.
“Hi, my name is Bobby Tables.”
AI: “Is that your full legal name?”
“Well, since you asked for it…”