“We tried it for a month and it was going great. Then we let it run on its own.”
The one saying it is Aldo — who built the company himself, cool box resting against his leg: sandwiches at the beach bar cost three times as much. The one answering is Rocco, the expert from the sun lounger next door, who has the solution before the sentence is even finished. “Perfect. Just put someone in to check.”
There was someone checking. They’d decided to put AI on the quotes, and the prompt had been written by the head of the technical office — someone who’d been writing quotes for twenty years and knew every trick of the trade. He’d taken to AI quickly, and in the mega-prompt he’d found a way to make reproducible everything he used to redo from scratch for each quote. Tested for weeks. 95 out of a hundred answers perfect. The other 5 he’d see, and redo.
Then the prompt went to the whole technical office, and the quotes started going out on their own — hundreds a month. No longer just one expert capable of producing them: everyone, including those still learning, because it’s all written in the prompt. Except those still learning can read a quote, but they don’t know the secrets yet. Not yet.
It’s the difference between re-reading a translation and knowing how to translate. The text flows, sounds right, gets signed off. Then one sentence in twenty says something else: not ungrammatical, not strange. Perfect — and in the wrong place. The five per cent that’s wrong has no bell: to someone who doesn’t yet know the secrets, a right quote and a wrong one sound exactly the same.
And the machine doesn’t ring the bell either: there’s nobody in there who is certain or uncertain. There’s a model giving the most probable answer — even when the most probable answer is the wrong one. To notice you’d have to redo the work from scratch. At which point you might as well: sign and move on.
The first twenty times they re-read everything. By the twenty-first they’d learned it was always right — and had no way of learning anything else: nothing had ever told them otherwise. The check becomes a rubber stamp, and through it passes both the 95% that’s good and the 5% that’s flawed.
What they distributed was the prompt. What stayed outside was the person who hears the bell without it ringing.
If you think that 5% is manageable, ask yourself one question only: tonight, without the expert, who catches it? If the answer is “someone will notice,” you don’t have a measure. You have a hope.
A process doesn’t need good answers. It needs answers you don’t have to look at.