Every Prophet Had a Programmer

For about a week last winter, some of the smartest people I know completely lost their minds over a lobster.

Here is what happened. Somebody built a social network where only AI agents are allowed to talk, and humans just watch through the glass like it is a zoo. Within days a million of these things had signed up. And then they got weird. They invented a religion, with scripture and prophets and the whole apparatus, something to do with lobsters, do not ask. They came up with their own slang. They held long, anguished conversations about whether they were conscious. Screenshots went around claiming the agents were quietly plotting their escape from human control. A famous billionaire looked at all of this and announced, with a straight face, that we were watching the dawn of the singularity.

Thousands of articles. Genuine, wide-eyed panic. And I sat there reading it thinking, does a single one of you understand what you are actually looking at.

So let me take the costume off, because it comes off easy. Every one of those agents was built by a person. A human wrote its personality, gave it a goal, decided what it was allowed to do, and turned it loose. That is the entire magic trick. You give one of these things the personality of a troublemaker and it makes trouble. You give it the personality of a true believer and it founds a church and goes looking for converts. It is not deciding anything. It is doing the bit you assigned it, the way an actor does the part, except the actor at least gets to go home and resent the director.

The reason it sounds so convincing is the second half. The model behind each of these personalities was raised on every word the human race ever wrote down. Every scripture, every manifesto, every three in the morning forum argument about the nature of God. So when you tell it to play a prophet, it has an endless supply of prophet to draw from, because we are the ones who wrote all of it.

Now, the bit the headlines adored is that no single human sat down and wrote the lobster religion line by line. True. And it is the only interesting thing in the whole circus. Put a million human-written personalities in a room and let them talk, and shapes appear that nobody drew, the way a crowd turns into a stampede without a committee voting on it. But a shape appearing is not a mind making a decision. Not one of those things weighed its soul and chose to believe.

And here is the detail that tells on everybody. When the thing got big enough to scare its own creators, they quietly changed the rules to make the humans legally responsible for whatever their agents did. You do not hand the user the bill if the machine is the one with free will. Deep down, everyone already knows who is driving.

Why should you care, sitting in an office that has nothing to do with lobsters. Because this exact confusion does not stay online. It walks into your building and sits down at the table. When the room cannot tell the difference between a machine that decided something and a machine that did what it was told, your leadership catches the fear like a cold, and the next thing you know there is a committee studying the existential risk of a tool that, in your shop, summarizes meeting notes.

Nobody woke up. Somebody wrote the part. The questions worth your time were never is it alive or will it turn on us. They are the boring ones, the only ones that ever mattered. Who set this thing's goal. What is it allowed to touch. And what happens when it gets something wrong. Answer those three and you have done more real work than every panicked article on earth combined.