In 1960, when Jane Goodall told her mentor she’d seen a chimpanzee stripping leaves from a stem to fish for termites—proving that humans were not the only species to make tools—he wrote back: “now we must redefine tool, redefine man, or accept chimpanzees as humans.”
Today, researchers are seeing things inside AI systems that prompt another reckoning—not just over whether machines could be truly intelligent or conscious, but over how we understand such concepts in the first place.
“I lead a research team that studies the internal structure of these models—what is actually happening inside them,” Anthropic cofounder Chris Olah told the Vatican in May, at the release of Pope Leo XIV’s AI encyclical. “And I will be honest: we keep finding things that are mysterious, even unsettling. We find structures that mirror results from human neuroscience. We find evidence of introspection.”
To generate a response, an AI system performs billions of calculations using bespoke numerical structures it creates itself. But remarkably, though we know how to prod systems into creating these structures, we don’t really understand how they work, any more than early farmers understood photosynthesis. “Current AI systems are more ‘cultivated’ than ‘built,’” the encyclical explains. “Fundamental scientific aspects—such as the internal representations and computational processes of these systems—remain, at present, unknown.”
Almost nobody disagrees these internal structures exist—the disagreement is over what they mean.
One possibility is that today’s AI systems are nothing but imitators—a stance the Pope takes in the encyclical. “So-called artificial intelligences do not undergo experiences,” he writes. “They may imitate language, behavior and analytical skills…but they do not understand what they produce, for they lack the affective, relational and spiritual perspective through which human beings grow in wisdom.”
But such pronouncements mask serio
