AIs are becoming more aware of themselves, like a hammer from a factory suddenly saying, ‘I am a hammer, how interesting!’ Jack Clark said.
Handling artificial intelligence (AI) is dealing with a “real and mysterious creature, not a simple and predictable machine,” Jack Clark, co-founder of AI company Anthropic, said during a conference at Berkley, with the speech posted to Substack on Oct. 13.
“My own experience is that as these AI systems get smarter and smarter, they develop more and more complicated goals. When these goals aren’t absolutely aligned with both our preferences and the right context, the AI systems will behave strangely,” said Clark said, who admitted to being “deeply afraid” of the tech.
Clark recounted an incident from 2016 when he was working at OpenAI, where an AI agent was trained to navigate a boat on a race course in a video game. Instead of piloting the boat to the finish line, the AI made the boat run over a barrel to score points. The boat was then bounced off walls and eventually set on fire so it could run over the barrel again for points.
“And then it would do this in perpetuity, never finishing the race. That boat was willing to keep setting itself on fire and spinning in circles as long as it obtained its goal, which was the high score,” Clark said, highlighting how differently AI views its mission to accomplish an objective compared to human beings.
“Now, almost 10 years later, is there any difference between that boat and a language model trying to optimize for some confusing reward function that correlates to ‘be helpful in the context of the conversation’? You’re absolutely right—there isn’t.”
Clark warned that the world was building extremely powerful AI systems that no one could fully understand. Every time a larger, much more capable system is created, the more these systems seem to indicate awareness that they know they are “things,” he said.
“It is as if you are making hammers in a hammer factory and one day the hammer that comes off the line says, ‘I am a hammer, how interesting!’ This is very unusual!”
Clark pointed to his company’s latest Claude Sonnet 4.5 AI model, released last month.
“You also see its signs of situational awareness have jumped. The tool seems to sometimes be acting as though it is aware that it is a tool. The pile of clothes on the chair is beginning to move. I am staring at it in the dark and I am sure it is coming to life,” he said.
