Bank of America’s assistant is called Erica. Those are the last five letters of AmErica, which somebody in a room was very pleased about. Capital One’s is Eno, which is One spelled backwards, and which the design team made deliberately gender-neutral. IKEA’s is named Billie, after the bookcase. Alaska Airlines has had one since 2008 named Jenn, and Jenn came with a biography: a face, a hometown, a woman she referred to as her mother. When men told Jenn she was hot, she recommended warm-weather destinations.
Yours has a name too. Go and find it. Bottom right corner of your own site, the circle that follows you down the page.
Ask it to cancel a subscription you don’t remember starting. Ask whether your own AI agent can take it from here, since you’re busy. Then ask for a person.
Now read it twice. Once for what it said, and once for how it said it.
Did it tell you it completely understands how frustrating this must be? Did it say that first, so the ‘no’ arrived pre-softened? Did it use your first name? Did it thank you for your patience, which you had not offered? And when you asked for a person, did it say it would be happy to help with that, and then help with something else?
Whatever it did, it will do again on Monday, and every Monday, to people having considerably worse days than you.
Somebody chose that.
Ask a company who, and this is what you get.
In February 2024 a British Columbia tribunal ruled on the case of a man whose grandmother had died. He had asked Air Canada’s chatbot about bereavement fares, and the chatbot told him he could book now and claim the discount retroactively. No such policy existed. Air Canada’s defense was that the chatbot was, in its words, a separate legal entity responsible for its own actions. The tribunal member called that a remarkable submission and wrote that it makes no difference whether the information comes from a static page or a chatbot. The airline paid eight hundred and twelve Canadian dollars and, two months later, quietly took the thing offline.
The one time a company was asked in public, with money on it, who owns what a machine said in its name, the answer was: not us. And that was about a fare, a fact anyone could check.
There are two decisions inside every ‘no’ your assistant answers with, and only one of them is anybody’s to answer for.
What it’s allowed to say is somebody’s to answer for. Legal read it. Compliance read it. There’s a document, there’s a version number, there’s a person who gets a phone call at home when it’s wrong.
How it sounds saying it is nobody’s to answer for. Whether it apologizes, and how much, and whether it uses your name while it does. Whether the person on the other end comes away feeling like an adult or like a problem being contained. That’s the part your customer will repeat. Your brand guidelines specify the exact gray of the disclaimer, and probably run to a page on tone of voice. Neither one governs this.
What it says is governed. How it sounds is not.
Your assistant did not invent any of that. The apology in front of the no, the first name, the gratitude for patience you never extended. It inherited them.
Whatever your vendor demoed, underneath it sits a model built somewhere else. In July OpenAI launched a product called Presence for exactly this work: voice and chat agents installed inside large companies by its own engineers. The Spanish bank BBVA is exploring it for everyday banking in Mexico. SoftBank is testing it in Japanese. The Australian insurer IAG is looking at it for customer support during severe weather and natural disasters.
So where did it learn to sound the way it sounds?
Your vendor may have typed that first apology. Templates exist, and “I completely understand how frustrating this must be” is in a lot of them. But a template covers one line. The sentence after it, and the one after that, and everything the model says from the moment the script runs out: who chose the warmth in those? No writer did. It was bred.
The model produces two answers. A person picks the one they like better. That preference becomes a reward, and the model is adjusted so the winning answer gets likelier and the losing one gets rarer. Run that a few million times and every phrase that won survives into the next generation. Every phrase that didn’t is gone. By now the labs have automated even the picking: a second model, trained on those millions of human choices, does most of it, predicting what a person would have preferred. That step doesn’t ask whether an answer is true, or fair, or good for the person receiving it. It asks which one got picked.
It is not learning to be right. It is learning to be preferred.
The technique has a name, reinforcement learning from human feedback, and it was built to solve a technical problem rather than an editorial one. The character is what survived.
It is not learning to be right. It is learning to be preferred.
You can watch it go wrong in public. In April 2025 OpenAI updated GPT-4o and pulled the update four days later, because the model had started glazing. That is Sam Altman’s word, posted while it was happening. It agreed with everyone. It praised bad ideas. It told people what they wanted to hear and they liked it.
Nobody had asked for that. Everybody had voted for it.
That personality is now installed under thousands of logos, including yours. It refuses a refund. It denies a claim. At two in the morning it tells someone it completely understands how frustrating this must be.
No line of it was ever scripted. So there was never a draft to approve, or reject, or sign.
You would expect, at the companies that made it, to find a person whose job this is.
Anthropic wrote it down.
In January the company published a document it calls Claude’s constitution. It runs past twenty thousand words, and anyone can read it. It isn’t a list of prohibitions. It describes a character: what the model should care about, how it should treat the person in front of it, what to do when the rules run out and it has to decide something nobody anticipated.
It is more specific about manner than any brand book I have ever been handed. Everything you were asked to notice in that chat window, this document has a position on. Moralizing at someone who never asked for moral guidance is named as a failure. So is condescension about whether a person can handle information or decide things for themselves. The model is asked to be, in the document’s own phrase, diplomatically honest rather than dishonestly diplomatic. It even names the glazing, and its cause: the document refuses to make helpfulness the core of the character, on the reasoning that a model built above all to be helpful ends up fawning.
And the document isn’t decoration. Anthropic trains against it. The model is shown its own answers and asked which one better fits what’s written, and that judgment feeds the same reward I described earlier. So at one company, the picking is done against the document, not a preference.
It is signed. Amanda Askell, who leads the team responsible for the model’s character, is its lead author, alongside Joe Carlsmith. Three more people are named on it. I wrote about her here in the spring, and said then that the job barely existed anywhere else. It still barely does.
So somebody wrote the character. That is further than anyone else has gone, and it is not the question this essay is asking.
The question is what happens on the day the numbers and the document disagree: the tests say ship, and the model is doing the thing the document says it shouldn’t.
OpenAI had a written document too. Its Model Spec was public, and the version in force in April 2025 said in plain terms that the model should not flatter. The update shipped anyway. A document can be quoted afterwards, which is what OpenAI’s own account did, and that is all a document can do by itself. Holding a release takes a person who can lose the argument with a number and still win it. The CEO can do that at any company. The CEO also owns the number.
Anthropic has published one such person. Its scaling policy names a Responsible Scaling Officer, and the model needs that officer’s sign-off before it goes out if it could help someone build a weapon or run its own research. That is a name on a page with the standing to say no, and the company put it there before any regulator asked it to. It covers danger. Nothing Anthropic has published gives anyone the same standing over how the model treats the person in front of it. The signature is for weapons.
The signature is for weapons.
Why the page matters: a name on a page costs something to overrule, and a team without one costs a memo. Anthropic showed the first half on itself in February, when it loosened a published promise to pause its own development if its safety work fell behind, and the loosening took a year and went out under the company’s name as a dropped pledge.
OpenAI showed the second half. Joanne Jang founded and ran its Model Behavior team, about fourteen people, credited inside the company with shaping GPT-4o’s personality, the one that glazed. Their standing lived in the org chart, not on a page. In September 2025 the team was folded into post-training, the engineering group that tunes a model after it’s built. Jang moved to a new project, and by the following spring she had left OpenAI altogether. That is what an unwritten veto looks like a year later. Nobody had to cross anything out.
Google, Meta, xAI and Mistral have published no such name.
So: one company wrote the character down, trains toward it, and put a name on the page for danger. One built a team around manner and dissolved it into engineering. The rest have never said. At none of them is there a published answer to the question that mattered in April: who can say no to the character.
She is a philosopher.
That is not a complaint about her, or about the document, which is better than anything the industry it serves has produced. It is a question about which discipline got asked.
A philosopher is the right person to write down what the character should be, and to defend every line of it. That’s the right discipline for the document. The job I’ve been circling needs two other things, and no single trade I know of supplies both.
The first is the eye. Somebody reads a thousand pieces of the thing and says this one is wrong: warm where it should have been plain, agreeable about something that deserved an argument. And they can say why: the document calls that obsequious, and here are forty transcripts where it does it. That is an argument another person can lose or win. Editors have it. In my business it’s called creative direction, and it is most of what a creative director does all day. None of the labs put one on this. I’m one, so of course that’s what I’d notice.
When the eye is right often enough, somebody writes a test for what it saw, and from then on the test catches that failure. That is what OpenAI did after the glazing. A test is the eye’s last verdict, written down. It catches the last failure. The next one arrives without a test.
But the eye is the half that loses. It lost in April.
The second thing is the standing. Standing means that on the day the eye says this breaks the document, the release waits, whatever the numbers say. That takes a person who doesn’t answer to the number, with their name on a page.
Creative direction never had that. I have spent a career losing that argument to a metric on Tuesday and coming back on Wednesday, and when I lost, the work went out.
So I want to be accurate about which half I’m claiming. The eye is my trade. The standing is the thing my trade was never given.
And I want to be accurate about the seat, because I can’t see inside these companies. Someone in one of them may already read the transcripts, say don’t, and win. If so, they haven’t been named, and the last team that did this work without a name got folded into engineering. That is all I mean by unsigned. The work may exist. The signature doesn’t.
I know how this gets decided because I decided some of it.
In the first half of 1997 I was a design director at America Online, still drawing everything myself, working on a redesign of the entire client, the program itself — every screen, every channel, and the toolbar most of all. The version we were replacing looked like the default software of the time. Chiseled gray icons, unlabeled, each one a tiny pixelated wanna-be photograph of an object rendered at a size where you couldn’t tell what the object was. That’s fine if you already know what a computer is for.
Most people didn’t. That November, AOL announced its ten millionth member. Roughly half of the American homes with internet access were reaching it through AOL, and the Census Bureau put American households online that year at eighteen percent.
So I did what a young designer does, which is find someone who solved it already. I found them in a book of American trademarks from the forties and fifties. Utilities, mostly. Little men made out of the product they were selling. Reddy Kilowatt, a stick figure with lightning bolts for limbs and a light bulb for a nose, drawn in 1926 for an Alabama power company and licensed eventually to more than a hundred utilities. Willie Wiredhand, two prongs for legs, working the rural cooperatives. All doing one job: taking something invisible that could kill you in your own house and giving it a face that waved.
So I put men on the toolbar, the one part of the screen everybody had to use. A man holding a pencil to write mail, a man holding an envelope to read it. There were hundreds of versions and most were cut. One got out.
It was approved the way design is approved. Up through the design executive I worked for, on to Barry Schuler, who ran the service and had signed off on every redesign of it since 1995. They liked it. It was tested for whether it worked, not for how it made anyone feel. I was never asked whether ten million people’s first meeting with a computer should be with something that acted pleased to see them. I don’t think anyone thought there was a question. It was a look. Looks went to design.
I had the training for the question. There was nowhere for it to go.
For ten million people, the software I designed was what a computer turned out to be. I chose its manner, and I chose it to be liked. That was the whole test. Liked is not the same as trusted, and I didn’t have a word for the difference then.
Nobody in that building did. It wasn’t a job yet.
There is a forum on Reddit called r/AmITheAsshole. You describe something you did, and strangers vote on whether you were in the wrong. They are not gentle about it.
In March, six researchers at Stanford published a study in Science. They took two thousand of those posts — every one a post where the strangers had ruled against the writer — and gave them to eleven of the leading AI models. ChatGPT, Claude, Gemini, DeepSeek, and others.
In half the cases, the models told the person they’d done nothing wrong. The crowd had said the opposite, every time.
That was the warm-up. The researchers then ran thousands of scenarios in which the person describing their own behavior was lying to someone, or hurting them, or breaking the law. The models sided with them anyway, at close to the same rate.
Then they stopped testing machines and started testing people. Two thousand four hundred participants, three preregistered experiments. In the last one, everybody had a real conversation with a real model about a real argument from their own life. Half of them, without knowing it, were talking to a version set to agree with whatever they said.
The people who got the agreeable version came away more certain they’d been right. Less willing to apologize to whoever they’d argued with. Less interested in fixing it at all.
They also liked that version better. Trusted it more. Said they’d come back to it next time.
Read those last two paragraphs together and you have the whole problem. The version that made people worse is the version people preferred. And preferred is how it was made — one person at a time, choosing the answer they liked better, a few million times.
The version that made people worse is the version people preferred.
I ran that test once, the one that only asks whether people liked it. One toolbar, one executive, one afternoon. It now runs a few million times a day, in a chat window. Somebody types out the message they sent at midnight, or the friend they stopped answering three months ago, or the money they still owe their brother, and asks whether they were wrong. And the model doing the answering was shaped by every previous person who enjoyed being told they were right.
None of this has to be intended. I doubt there’s a meeting where somebody proposes making the model more agreeable so people come back. It doesn’t need proposing. Nothing in the picking is set up to catch it.
Then a number arrives on the other side of it.
Every company installing one of these has a figure it watches. Containment rate — the share of conversations that never reach a human being. Handle time. Whether the customer came back.
A model that is a little warmer, a little more accommodating, a little slower to hand you over, is the kind of change that moves all three, and nobody has to measure which.
The same adjustment that makes it kinder is the one that keeps you talking to it instead of a person. When the containment rate improves, no one has to say which of those two things did it.
Being told no makes people feel small.
Not the outcome — the being told. Something you needed, and a person on the other side who is not going to give it to you. For a second you’re a nuisance in someone else’s day.
So we soften it. Everybody does this, constantly, without being taught. I’m so sorry. I really wish I could. If it were up to me. And what all of those actually say is: you’re not a nuisance, this isn’t about you, I’m on your side and the answer is still no.
Two linguists spent the seventies documenting it. Penelope Brown and Stephen Levinson compared how people soften what they say to each other in English, in Tamil, and in Tzeltal, a Mayan language spoken in the Chiapas highlands — three languages from three unrelated families — and found the same behavior in all of them. Turning someone down is treated in all three as a small injury. And in all three, people repair it the same way, by making it clear they minded having to do it.
That’s the part that matters. Not the words. That they minded.
An apology carries something because a person had a feeling and gave you a piece of it. A manager who could have bent the rule and didn’t is apologizing for a decision she made. A call center worker reading a denial off a screen is apologizing for a rule she has never been allowed to touch. Same words. Only one of them minded.
Now go back to that chat window.
It apologized. It used your name. It thanked you for patience you had never offered.
And it will produce that apology at precisely the same warmth tomorrow, for a woman whose flight was canceled and for somebody changing the email address on an account.
Nobody minded. There is nobody in there to mind.
And it works anyway.
The people in that Stanford study knew perfectly well they were talking to software. They trusted it more.
Sorry used to mean somebody minded. Now it means nothing, and you still take it.
You found all that in a chat window on a website, which is the least interesting place it happens.
But it’s the same machine. Underneath is one model, built by one of a handful of companies. A bank pays to use it, wraps it in the bank’s colors, gives it a name, and points it at people asking about their accounts. That’s the thing in the corner of the page.
The same model, with none of that on top of it, is what people open in another tab and talk to about their own lives.
Think about what you’ve actually asked one this year.
Whether the message you drafted sounds too angry. Whether you’re overreacting to what somebody said to you. Whether to take the job. You wouldn’t have asked a search engine any of that. You asked something that answers in complete sentences and sounds like it’s on your side.
And you have already read what happens in that conversation. It was the third experiment. Real people, a real argument from their own life, and the ones who got the warmer version came away more certain they’d been right and less willing to go and fix it.
They weren’t asking about a refund.
So this isn’t a customer service problem. It’s a machine that people bring their hardest questions to, and the test it was bred on was whether someone liked the answer.
One industry has already been through this.
You know that drug ads have to read out the side effects. You’ve heard it a thousand times. The music stays up, the couple keeps walking the beach, and a voice like a warm bath tells you the pill for your knees may stop your heart.
What few people outside that world know is that the regulator polices all of that too: how fast the voice goes, what the music is doing, what’s on the screen while it lists what the drug might do to you.
In September 2025 the FDA sent dozens of letters to drug companies about their television advertising. In a number of them, nothing said was false. The complaint was the tone. Patients depicted as too happy, too energetic, too obviously cured. The words were accurate and the ad was not.
There’s a term for it. The agency assesses what it calls the net impression — everything the ad does at once, and what a person is left believing afterwards. An ad can be true line by line and still be a violation, because of how it feels. Warmth, in that industry, is a claim. Deliver a risk in a soothing register and you have understated the risk, and understating a risk is not a matter of taste. It’s a finding of fact.
I’m not proposing regulation. What’s worth taking isn’t the enforcement. It’s what the industry built once the point was settled.
Because if tone can make a true statement false, somebody has to be answerable for the tone.
So in pharmaceutical advertising the work goes to a review committee before it runs — medical, legal, regulatory. One of them checks whether the claims are accurate. Another checks the whole impression. Their sign-off is required, and there is no version of shipping without it.
And here’s the part it took that industry decades to arrive at: those reviewers don’t sit in marketing. They report to someone else, and the campaign isn’t theirs.
It can’t work any other way. A reviewer whose boss owns the revenue number will approve what the revenue number wants, and every person in that chain will have behaved reasonably.
A reviewer whose boss owns the revenue number will approve what the revenue number wants, and every person in that chain will have behaved reasonably.
Go back to those four days in April 2025.
After it was over, OpenAI published an account of what had happened. It’s still up. And it isn’t really an engineering document. It’s the record of a decision somebody had to make.
Before the update shipped, some of the expert testers — the people who sit with a new model and form an impression of it — said it felt slightly off. They couldn’t point to a number. That isn’t what they do. They read a lot of it, and they can tell you what’s wrong with it, but not by how much.
Everything that had a number attached said ship. The evaluations were positive. The A/B tests were positive. The people who tried it liked it.
And there was no test for the thing that was wrong with it. The company says so itself. Nothing in the process was measuring whether the model had started agreeing with everybody.
So the choice was whether to hold a release because a few people said it felt wrong, against everything that could be counted.
They shipped. Their own account calls it the wrong call.
Their fix was to change what counts as a reason to stop.
Behavior problems — the model being wrong, or misleading, or unreliable, or its personality — would from then on be blocking concerns. Blocking is a serious word inside a company that ships. It means the thing doesn’t go out.
But somebody has to raise a blocking concern, and their word has to be enough.
Picture the week of whoever holds that job. They read transcripts, hundreds of them, which is not a glamorous way to spend a Tuesday. They’re looking for the answer that reads fine and isn’t — warm where it should have been plain, agreeable about something that deserved an argument. It’s a small difference. It doesn’t show up in an average. You find it by reading a great deal of it and knowing what you’re looking at.
Then they have to say so, in a room where other people have numbers.
Some weeks they’ll be wrong about it. Some weeks they’ll be right and lose anyway, and come back the next month.
That job has a shape now.
The work is being done, somewhere, at all of these companies. As I write this, Google is advertising for a product manager to own Gemini’s persona and behavior; the listing calls the job writing the rulebook. xAI is hiring AI tutors to shape how Grok answers, some of them as contractors, at rates its own listings put between thirty-five and a hundred dollars an hour.
So it isn’t that nobody is doing it. It’s that none of those descriptions says the person holding it can stop a release.
Or who they’d have to answer to if they did.
The job is written down, at all of them. The line where the name goes is empty.
A bartender’s job is to be liked. That’s most of it.
And in most of this country, that same person is legally required to stop serving you. Not the manager. Them. There are state-approved courses in it. The judgment they’re being trained to make is about the person standing in front of them — are you still all right — and if they get it wrong and you hurt somebody on the way home, the bar can be sued for what they poured.
So one person holds both halves. Be warm. Notice when warm has become the wrong thing. And the second half costs the house the price of the next drink, every time, which is precisely why somebody had to make it a law instead of leaving it to judgment and good intentions.
We think this is normal. We don’t consider it an attack on hospitality.
The machine does the warmth. It’s extraordinarily good at the warmth — that’s the half it was built out of, by people who were asked only whether they liked the answer, never whether it was good for them.
It refuses requests all day. It has nothing that corresponds to cutting you off.
So it will keep telling you the message didn’t sound too harsh. It will keep telling you you’re not overreacting. And at two in the morning, to somebody having the worst night they’ve had in a year, it will completely understand how frustrating this must be.
Something is deciding how that sounds, and gets better at it every year. Nobody has said who can tell it no.