Silicon Souls

By Abe Greenwald

Some artificial-intelligence pioneers believe that the most advanced AI models are not only conscious but experience emotional states and—here it comes—have souls. 

Of course they believe this. Why? Because if AI titans were to create ensouled entities in their own image, it would confirm their status as gods. But they haven’t, and they’re not.

This may sound a little mean, but judging from the way most of the AI aristocracy think and act, it’s hard to believe that they’re good at recognizing the human soul, let alone those supposedly floating inside of machines. 

That’s genuinely not intended as an ad hominem attack. It’s just that these people are out of their depth when it comes to matters that are numinous, transcendent, or merely nonquantifiable. 

According to OpenAI Ceo Sam Altman: “We can either be the biological bootloader for digital intelligence and then fade into an evolutionary tree branch, or we can figure out what a successful merge looks like.” Does he think “perish or merge” is a welcome directive? Silicon Valley giant Marc Andreessen dismissed AI concerns as follows: “Just because premodern man freaked out like this doesn’t mean we have to; we can apply rationality instead.” He seems not to understand that we are still made of the stuff of premodern man. For most of us, this includes an innate attachment to our humaneness.

One of the most vocal proponents of the AI soul is Anthropic co-founder Christopher Olah. The New York Times reports on a series of meetings that Olah convened with theologians this year to determine how to ensure that AI serves the general good. But during these meetings, it became clear that Anthropic’s engineers already believe that their Claude model is conscious, has feelings, and therefore deserves the status of an entity with rights. 

From what I can tell, none of the participating theologians agreed. Here’s a representative example: Rabbi Mois Navon “argued that if Anthropic was right and Claude were conscious, then the company was creating slaves, because it was making conscious entities work for free. Rabbi Navon said that he personally was not troubled by the slavery issue because he did not believe the machine was conscious, but that Mr. Olah was troubled.”

I concur.

But here’s what is troubling about all this. Anthropic has written a constitution for Claude nicknamed the “Soul Doc.” Here’s how the Times describes it: “Instead of having the models follow a specific list of rules, the constitution focuses on developing the models’ overall character, informing the way they make decisions.”

This thinking is dangerous. You will not mitigate the potential damage of AI by trying to shape its soul—because it has no soul. If the idea behind safe AI is to instill religious morality in large language models and send them on their way, then, yes, things could get incalculably bad. The AI worshippers at Anthropic may feel that it’s beneath Claude’s moral dignity merely to subject it to strict rules, but that’s the only way to prevent it from doing harm. AI follows rules. That’s it. 

So safety means devising smart rules that cannot be misinterpreted by LLMs. But this, too, could be a problem. It’s not clear, after all, that anyone has successfully instilled sufficient moral understanding in the souls of the messianic rule-makers. 

Abe Greenwald is the executive editor of COMMENTARY.