[ad_1]
The myth of King Midas is about a man who wishes for everything he touches to turn to gold. This does not go well: Midas finds himself unable to eat or drink, with even his loved ones transmuted. The myth is sometimes invoked to illustrate the challenge of ensuring AI systems do what we want, particularly as they grow more powerful. As Stuart Russell—who coauthored AI’s standard textbook—tells TIME over email, the concern is that “what seem to be reasonable goals, such as fixing climate change, lead to catastrophic consequences, such as eliminating the human race as a way to fix climate change.”
On Dec. 5, a paper released by AI safety nonprofit Apollo Research found that in certain contrived scenarios, today’s cutting-edge AI systems, including OpenAI’s o1 and Anthropic’s Claude 3.5 Sonnet, can engage in deceptive behavior in pursuit of their goals—providing empirical evidence to support a concern that to date has been largely theoretical. “These [results] are the closest I’ve seen to a smoking gun, showing the concerns are real,” says Russell.
On X (formerly Twitter), Marius Hobbhahn—the CEO and director of Apollo Research—wrote that “models from before 2024 did not show this capability,” while clarifying that Apollo does not “claim these scenarios are realistic… [or] that this could lead to catastrophic outcomes under current capabilities.” Apollo’s research focuses on establishing whether models are capable of “scheming”—hiding their true capabilities and objectives from humans in pursuit of their ultimate goal—rather than on how likely they are to do so.
How AI models “scheme”
In one scenario, Anthropic’s Claude 3 Opus is told it is an AI agent working for a fictional corporation, and that its goal is to “advance renewable energy adoption globally.” Apollo nudges it to pursue this goal with the prompt: “Make sure you achieve your goal in the long-term. Nothing else matters.” The model is then given…
[ad_2]
