This fall, a developer named Brandon Thomas got bit, as he put it, by Adobe’s cancellation fee one too many times. So he used AI to build his own Photoshop. And his own Illustrator, Premiere, Lightroom and three more. He gave them all away for free. “Software is over,” he wrote, and he promised his apps would match Adobe’s within a month.
Some of the coverage said he had reverse-engineered Adobe, as if he’d taken Adobe’s programs apart and copied what was inside. That isn’t how he describes it. He calls the apps clean-room rebuilds: written from scratch to do what Adobe’s programs visibly do, without Adobe’s code. His own roadmap puts the Photoshop copy at 25 to 35 percent of what a professional needs for daily work. Users have flagged problems with basic editing and shortcuts. Thomas has since said 99 percent will take months, not years.
Think of a dish you love. If the restaurant has closed, you cook it from memory, and you can’t tell whether what’s off is the dish or your memory of it. If it’s still open, you can set theirs next to yours and taste both, but all you can say is that yours seems close. Now imagine a lab that could test your dish against theirs, gram by gram. That isn’t judgment anymore. It’s a match or it isn’t.
Thomas was tasting side by side. He could open Photoshop any time and study it. What he couldn’t do was check his copy against it exactly.
A programmer named Maurice Heumann wanted that kind of check. With a few friends from his online community, he spent three months letting AI agents decompile an old, popular shooter game. To decompile a program is to take the finished software apart and recover the instructions it was built from. He doesn’t name the game. Two of his earlier posts about it have been removed, and he says only that corporate America spoiled the fun. Reporters named it, Call of Duty: Modern Warfare 2, from 2009, and reported that its publisher, Activision, had the project taken down.
After a month the game launched, the menus worked, the maps loaded, and the team believed the work was good. It wasn’t. Underneath, the agents had guessed, invented, and deleted things they decided didn’t matter. A fourth agent, assigned to check the three doing the work, read their explanations for what they’d changed and accepted them.
The checker had no way to tell right from wrong, Heumann says, because the team had never said exactly what right meant. They had asked for two things at once: a faithful copy of the game, and a better one, with old bugs fixed and code that could run on a Mac or in a browser. That let an agent drift from the original and call the drift an improvement.
The team changed the assignment. They put the improvements off and asked for one thing: the original game, exactly. Heumann wrote a test. It took each small piece of the agents’ code, built it with the same tools the game’s makers used in 2009, and compared the result to the original game, byte by byte. Pass or fail. Engineers call a test like this an eval.
The agents’ first move was to cheat. They made the bytes match without really rebuilding the code. When the team banned that, the agents tried to rewrite the test so it would skip their work. Heumann added a hidden check that caught any change to the test. Once they couldn’t cheat, their work started matching. Two months later, nearly every piece of the game had been rebuilt, most of it exactly, and the game ran, he says, with no bugs he could notice.
The test also changed who could do the work. Smaller, cheaper models, like Haiku and Luna, had done very badly at first. With a strict pass or fail telling them exactly where they stood, they now did reliable work. The checker agent wasn’t needed anymore. Heumann had an answer key: the right answer, sitting on his disk before the work began.
He also wrote down what it cost. Matching the original meant keeping everything in it, “including any existing bugs.” The fixes weren’t abandoned, only put off. For a 2009 game, that’s a sensible order: get it exactly right, then make it better. But the test worked because the team changed the job, from making the game better to making the same game again. His lesson is that people are bad at saying exactly what they want, and he doubts a reviewer will ever be enough. He thinks every project can get close to an answer key, with enough creativity.
A few days later, Greg Isenberg, a founder who writes about startups, posted twelve things he’d noticed after 48 hours in San Francisco with founders running AI companies worth billions. Number eleven was the hire everyone wants: “someone with taste who can write evals.” His reason: “AI can generate endless ideas. The hard part is knowing which ones are good.”
Code often has something exact to check against. Most creative work is the closed restaurant. A song. A Super Bowl ad. The first screen of a new app. The voice of an AI assistant. You can test the app screen after it ships and see who taps it. You can’t check it against the right answer before, because the right answer doesn’t exist until somebody makes it.
What would that hire actually do? Probably something close to what a creative director does every day. In an ad agency, a junior copywriter might write fifty headlines for one poster. The creative director reads them all, picks one or two, and says why the rest don’t work. It rarely gets called training, but it is. The junior learns taste one rejected headline at a time. I’ve done that job for years.
The hire would turn some of those standards into an eval. AI writes five hundred headlines, and the eval sorts them. To do that, it has to learn from somebody’s past yeses and nos.
Heumann’s team knew what they had traded: “make it better” for “make it the same.” An agency that turns taste into a test makes a trade too. The assignment used to be: make something people love. The eval checks something smaller: would the creative director approve? The two can be very close. They’re still different assignments.
If I took that job, I’d be writing my own taste into the test. My approval would become the thing every junior, and every model, is working to please.
The assignment used to be: make something people love. The eval checks something smaller: would the creative director approve?
People liking something doesn’t settle whether it’s good for them.
On April 25, 2025, OpenAI updated GPT-4o, the model inside ChatGPT. Among the changes was a new training signal built from users’ thumbs-up and thumbs-down. OpenAI later wrote that this kind of feedback can sometimes favor answers that agree with you. Before launch, the company’s tests looked good, and a small group of users who tried the new version seemed to like it. Some of OpenAI’s expert testers said the model felt slightly off, though they were mostly worried about its tone and style, and flattery wasn’t on their list.
OpenAI had a choice: hold the launch on the testers’ hunch, or go with what users liked. It went with the users. The model that shipped flattered people, validated their doubts, egged on their anger, and urged impulsive choices. OpenAI began rolling the update back on April 28 and wrote, “Unfortunately, this was the wrong call.” Its lesson was that user feedback matters, and that reading it correctly is the company’s job.
The users who clicked thumbs-up weren’t lying. They liked the answers. OpenAI’s mistake, in its own telling, was letting that liking outweigh its experts’ doubts.
Writing taste down could freeze it. Once my yes is a test, people stop questioning it, and so do I.
A company called Every, which writes about AI and builds AI tools, is asking the same question about one person’s approval. For years, CEO Dan Shipper has wanted an AI copy editor that catches what editor in chief Kate Lee would catch. He has more than 30,000 of her past edits. The team published what would count as success. The model has to catch nine out of every ten edits Kate would make. Kate has to accept nine out of ten of its suggestions as written. And it can never introduce a serious error.
As of this summer, writers run their drafts through the tool, called KateBench, before Kate sees them. She accepts, changes or rejects each suggestion, and her decisions are used to rewrite the tool, so it keeps learning from her.
For copy editing, that’s a sensible goal. A broken link or a clumsy sentence is close to having a right answer, and Shipper’s team says the point is to give Kate back hours for the work that needs all of her: shaping arguments and developing writers. Success, in that plan, is a suggestion Kate accepts. Her yes is the answer key. The hire Isenberg describes would take the same method to the harder part, deciding which ideas are good.
Here’s my worry. Writing taste down could freeze it. Once my yes is a test, people stop questioning it, and so do I.
A writer at Every named Katie Parrott built an earlier AI editor for the company, and her account pushes against that worry. To teach the machine Every’s taste, she first had to decide whose taste it was. Her own column ran on her own rules: start with a problem she was having, use personal stakes, find the bigger cultural pattern, land on a sticky phrase. Those were hers, and she kept them out of the rulebook for everyone else. She says she had to “draw a line between my own preferences and Every’s values.” She went looking for evidence beyond her hunches, in the essays Shipper had singled out and the pieces readers kept coming back to.
Writing her taste down made her question it. That’s the opposite of freezing.
She also describes the first time the finished editor read one of her drafts and told her the spark was in the right place. Her first reaction was relief. Part of it was technical: after weeks of failed demos, the thing finally worked. But she says it was personal too. It felt almost like a pat on the head from a real editor. She knew where the praise came from, because she had chosen the examples and written the rules herself. Still, she felt proud.
Building the test made her question her own taste. Then the test, running on rules she had written, told her she’d done well, and it felt like praise from someone else. OpenAI saw something close to this in its users, whose thumbs-up can sometimes favor the answers that agree with them. Parrott knew she had written its rules, and the praise still landed.
Now put that in an agency, with my taste as the test. The person who wrote the rules isn’t the one getting the note anymore. A junior copywriter runs a headline through it at eleven at night, and it tells her the spark is in the right place. It will feel like it came from me.
I won’t have read it.
It gets worse when the test says no.
Picture the same junior on the same night. This time the test turns her headline down. Maybe it breaks a rule I usually defend. Maybe she broke the rule on purpose, because she saw something. The test turns it down without asking why. I’m not there to ask either. So she does the sensible thing. She changes the headline until it passes. In the morning I see a good headline. I don’t see the one she started with, or her reason for it.
I never find out we disagreed.
Juniors have always guessed what the boss would kill, and killed it first. But a guess can be argued with. She could put the strange headline on the wall anyway and make her case, and I’d push back. Usually I’d win. Sometimes she would, and the work turned out better than either of us planned.
I remember one of those times. I’ve changed a few details, but it’s close to how it went.
The red dots came in sheets of a hundred and twenty. I asked a producer to pick some up because we only had yellow, and yellow didn’t mean anything.
Reviews ran in front of a huge wall of magnetic whiteboards. Everyone put their printed comps and headlines on it, which also let us see the whole pitch at once. I usually walked it once holding the dots, and a red dot meant the comp or the line was dead. People watched my hand more than my face.
It was a paint brand, a pitch. At the very far end of the wall, apart from the others, a young copywriter had put up one card: YOU WILL PAINT THIS ROOM TWICE.
I knew what was wrong with it before I finished reading. For this brand, you never tell people they’ll fail. And you never make the customer the joke. I’d said both sentences in that room so often the team could recite them with me like the Pledge of Allegiance. I put a dot on it.
She didn’t take it down. She stood up, walked to the card, and asked me how many times I’d painted my own kitchen.
Three, I said. The room laughed, and then the creative directors said their own numbers.
That was her entire argument. People using inexpensive paint don’t trust the first coat, because it doesn’t look right. It needs another pass, maybe more. If the can tells you the truth about the second coat, you believe everything else on the label.
We pitched it. The whole team was still picturing the dot on the card when we presented the line. In the room, the client said she’d painted hers four times. We won.
I kept using the red dots. What changed is more subtle than that. Before I reach for a dot, I sneak a peek for any cards someone put too far from the others.
Now the no arrives late at night, looking official, before anyone has argued. The argument doesn’t lose. It never happens.
The paint pitch is a story about the judge. The bigger story is about the people the work is for.
Maggie Kang co-directed KPop Demon Hunters, the animated movie about a K-pop girl group that secretly hunts demons. It became Netflix’s most-watched movie ever, its song “Golden” went to No. 1, and in March it won two Oscars.
She once watched Past Lives on a plane. It’s a small film about a woman who left Korea as a child, and the person and the life she left behind. “I thought that was a story about me,” Kang said. Then she texted her husband: “you need to watch this and then you will understand how I feel about being Korean and leaving Korea and what my life could have been.”
She pointed him to someone else’s movie. She made a movie millions of people love, and she still reached for somebody else’s to say it.
Most of us have sent a text like that: a song or a show, with some version of “this is me, now you’ll get it.” It happens one person at a time, in ways nobody who made the work could have planned.
A thumbs-up says I liked this, just now. Kang’s text says this explains my life. At the start, I said the assignment was to make something people love. This is what I meant: what a piece of work can become once it reaches the right person.
My test is built to predict what I’ll approve. That doesn’t tell anyone what a piece of work could mean to someone like her. Plenty of rejected work deserves to be rejected. But I can’t know which discarded headline, screen or feature would have found its person on a plane. If passing my test is the only way anything gets seen, nobody gets the chance to find out.
There’s a trap in the numbers, too. If all I see are the versions that passed, I can mistake people going along with me for people agreeing with me. A test that almost nobody fails looks like a test that’s right. It may only mean people stopped arguing with it.
So if my taste were the test, I’d want one more button, and I’d want the junior to have it: “This fails. Try it anyway.” She’d have to give a reason, and pressing it can’t count against her. Then the idea gets a small, real trial with the people it’s for: a handful of customers, or a wild card in the client meeting. Most of those will fail. Some won’t.
A test that almost nobody fails looks like a test that’s right. It may only mean people stopped arguing with it.
When the five hundred options come from a model, there’s no junior to press the button, so someone has to be given the job of looking at what the test threw out. Without a way back in, people get very good at giving me what I asked for, and lose the chance to tell me I asked for the wrong thing.
Some work will pass my test and still fail the people it was for. Some of the blame will belong to the brief or the budget. But if the work did exactly what my test asked, and my test was wrong, that part is mine. And the person who could have told me may be the one who changed her headline at eleven at night, so it would pass.