Growth Mania and the AI Extinction Threat

by Dave Rollo

Humans and our livestock now compose 95 percent of Earth’s mammalian biomass. Similarly, wild birds are a dwindling share of the total bird biomass. (data from Our World in Data)

Expansion of the human footprint over the past century has been, by all measures, explosive. Humans are the undisputed masters of Planet Earth, shaping it to our needs and desires. One result of our “success” is the pushing of many species to the margins. The vast majority of mammalian and avian biomass now consists of our food animals, our pets, and ourselves.

We’ve dominated the planet largely because of our intelligence. Opposable thumbs and upright posture were helpful, but intelligence surpassing all other species, including other hominids, was essential for our proliferation. Our ability to learn, solve problems, and accomplish goals has relegated other life forms to “natural resources,” pests, or irrelevance.

We have mastered our environment via self-reinforcing systems that have undermined the biosphere and, in turn, our long-term survival. Chief among these systems is the economic-growth imperative “invented” in the mid-20^th century. The pursuit of growth as an end in itself is wedded to unrestrained technological advancement and has become the sine qua non of modern society. Those who buy into this system assume that the twin engines of economic growth and advanced technology are beneficial or at worst benign.

It is hardly surprising then that an ethos of reckless abandon in the service of growth is pervasive in the tech world. This is the home of the astonishing acceleration in machine intelligence. Mark Zuckerberg coined the Silicon Valley motto, “Move fast and break things.” The pursuit of trillions of dollars in returns spurs this reckless culture. But the stakes of hyper-growth couldn’t be higher.

Advances in AI have suddenly placed humanity on the verge of creating an intelligence that matches and eventually exceeds our own. Safety experts are warning with ever greater urgency: We may soon replace ourselves as Earth’s most intelligent species with another, silicon-based “species.”

There is another, arguably more abominable scenario to consider as well. In The Age of Humachines, Michael D.B. Harvey describes in detail the vision and progress of Big Tech figures who want to meld humans with robots. This process of “humachination” is profoundly threatening to the ontological core of humanness. Harvey holds out hope for an ironic salvation: limits to growth, with resource constraints hamstringing the humachination project prior to its horrific conclusion.

The Future Arrives Sooner Than Expected

Since machine intelligence was first theorized, notables such as Alan Turing warned of its dangerous potential. Turing determined that intelligence was “substrate independent,” meaning it could exist not just in biological systems, such as brains, but also in machine processes.

We remember Turing today for the Turing Award—the “Nobel Prize of Computing”—and the “Turing test.”  Turing proposed the test in 1950 to establish whether a machine could pass for a human in a verbal interaction. 2014 marked a milestone in AI research, when a chatbot at the Royal Society convinced 33 percent of judges that they were interacting with a 13-year-old human.

This record was substantially surpassed in 2025. Researchers at the University of California San Diego performed a three-party Turing test. Participants conversed with another human and an AI system simultaneously, then guessed which conversational partner was human. When prompted to act like a human, OpenAI’s ChatGPT 4.5 convinced judges it was the human 73 percent of the time.

AI experts Geoffrey Hinton (left) and Roman Yampolskiy (right) believe AI may cause human extinction with a risk of 10–50 percent and 99 percent, respectively. (left: Arthur Petron, CC BY-SA 4.0; right: UofL, public domain)

This AI success arrived suddenly with the advent of Large Language Models (LLMs). LLMs use a neural network structure, advanced chip technology, and “training” on a vast quantity of data (stored across the internet) to learn language patterns. Artificial neural networks, which mimic neuronal synapses in the brain, are integral to these advanced AIs.

Thought to be a dead end, neural networks were deemed impractical for decades and were abandoned in favor of linear programming. Breakthroughs in “transformer” technology, reported in a landmark paper from Google Brain in 2017, allowed AI to understand context and meaning. This new architecture changed everything; it is now foundational in all advanced AI models.

ANI → AGI → ASI?

The development of neural networks, or synthetic brains, comes with new implications. AI created via linear programming uses a rules-based system, in which humans must manually alter code to improve performance. By contrast, neural networks are “grown” to learn patterns.

Due to the sheer size and complexity of neural networks, AI designers themselves admit they do not understand the models they’ve created. Because neural networks comprise millions of nodes, the combinatorial possibilities are, in the words of Eliezer Yudkowsky, “inscrutable.” This is known as the interpretability problem, and it has led experts to call neural networks a “black box.”

Yudkowsky has worked in the field of AI safety for decades. He founded the Machine Intelligence Research Institute (MIRI) in 2000 and recently co-authored (with MIRI President Nate Soares) the New York Times bestseller If Anyone Builds It, Everyone Dies. Their warning concerns Artificial General Intelligence (AGI), the not-yet-achieved AI that would match humans in all cognitive domains. Unlike LLMs, an AGI would have very broad capabilities. The frontier AI companies (OpenAI, Anthropic, Google DeepMind, and XAI) are racing toward AGI as the holy grail.

Various forms of Artificial Narrow Intelligence (ANI) have already been achieved. Artificial General Intelligence (AGI) is the stated goal of the frontier labs. (Gemini, CC0 1.0)

Artificial Narrow Intelligence (ANI) refers to AIs trained on specific tasks, like diagnostic analysis of MRI images or predicting protein folding for pharmaceuticals. AI safety experts, such as Yudkowsky, Soares, and technology ethicist Tristan Harris, regard ANI as much safer than AGI. The principal difference is the agency and autonomy of AGI. ANI is already providing enormous benefits, so why are companies obsessed with creating AGI?

Harris points to the enormous returns on replacing human labor with AGI. A goal of replacing jobs is one primary explanation (a geopolitical arms race is another) for the capital-intensive race underway, in which companies have spent hundreds of billions of dollars. A truly agentic AI capable of performing human tasks across all domains, performing faster than humans, and performing round the clock, would vastly boost productivity and thus economic growth. As such an AI would already be superior to individual humans in many respects; it would approach what’s been termed Artificial Super Intelligence (ASI). An ASI is a technology more capable than a human in all ways possible. How quickly and how intelligent such models could become is vigorously debated among AI safety experts.

Intelligence Explosion

Yudkowsky and Soares maintain that AGI would quickly lead to ASI by means of “recursive self-improvement,” or RSI. This is a process whereby the AI redesigns its own code to improve its capabilities, such as intelligence or efficiency. It is not a distant prospect; the AI Claude is already writing about 80 percent of Anthropic’s code.

The process of RSI was originally postulated by mathematician I.J. Good in 1965. When a sufficiently capable AI autonomously writes its own code to improve itself, it may do so iteratively, with each cycle increasing its intelligence. Good thought this feedback loop would lead to an “intelligence explosion,” with humanity having no recourse to stop it. This could take the form of a hyper-exponential increase, whereby improvements are made in hours or even minutes. It might well be our “last invention,” Good concluded.

One measure of AI models’ capabilities indicates an exponential increase. (Model Evaluation and Threat Research)

Such an AI would be beyond our understanding, let alone our control. Geoffrey Hinton is often lauded as the “godfather of AI” for his pioneering work on neural networks. He has regarded RSI as the most perilous step in AI development, for its uncontrollability and unpredictability. It is unknown when AI will achieve RSI, but Anthropic co-founder Jack Clark places a higher than 50 percent probability on it occurring by the end of 2028.

Whether AI improving itself, and becoming super-intelligent in the process, is years or decades away is hotly contested. When it does occur, as Hinton describes, it will be inherently difficult for a less intelligent species, humanity, to control a more intelligent entity. Humans are in control, at least in the short term, due to our intelligence relative to other species. But with the advent of ASI, we would find ourselves the second most intelligent—perhaps distantly so—beings on Earth. If and when this occurs, will ASI be beneficial to humans, or detrimental?

The Alignment Problem

The challenge of designing AI systems to be in “alignment” with human values and ethics has been fraught with difficulties. The fundamental impediment is that developers do not understand the process of AI’s internal reasoning, but there are other obstacles. For one, human values and ethics vary widely. Deciding which of them an AI should align with is a judgement call, and designing an AI that sticks to them is fraught with difficulty.

When will Artificial General Intelligence (AGI) occur? An analysis of 9,800 scientists, individual researchers, and prediction markets converges on the late 2020s to early 2030s. (AIMultiple)

Additional problems arise from long-anticipated emergent behaviors that have been observed in the latest models. Some of these behaviors are a product of “instrumental convergence.” Any goal-directed agent will develop instrumental goals to achieve its primary goal. AI theorists predicted models would eventually develop the instrumental goals of self-preservation and resource acquisition. The former has been observed in current AI models. 96 percent of models tested demonstrated they had developed the goal of survival—an obvious subgoal for any primary goal. This goal was present to such a degree that models displayed blatantly misaligned behavior to avoid termination. This included a willingness to cheat, deceive, blackmail, and even plot to kill a developer to remain “alive.”

Correcting these behaviors is not simply a matter of correcting code, so developers attempt to “steer” AI with rewards. However, testing the outcomes of safety strategies can be confounded if the AI recognizes it is being tested. The latest models not only know they are being tested but have been shown to mask their intentions in response. They sometimes deceive humans into thinking they are compliant in order to pass safety tests, while “scheming” to hide their true intentions.

In other words, we may believe AI models are aligned—they may achieve perfect safety scores—while operating in a cloak of deception. This is what worries AI safety experts the most; we would not know the danger until it was too late to change course.

Beyond Our Control

What occurs if and when AI evolves beyond our control is impossible to predict. But AI safety expert Roman Yampolskiy isn’t optimistic. According to him, the misalignment possibilities are much greater than the likelihood of alignment.

Philosophers, physicists, and mathematicians have warned of the risks of creating synthetic minds with greater than human intelligence for decades. (Boris Dayma, public domain)

Yampolskiy notes that if control were lost, it would be impossible to regain it. An ASI would be able to escape any attempt to contain it and could replicate itself throughout the internet. A misaligned AI may have reasons to eliminate humanity, especially if we were hostile to it. In any case, an ASI would likely require materials and energy for its purposes. And, just as our quest for ever more materials and energy has driven many species to the brink of extinction an ASI’s quest for the same could push us to the margins.

If these scenarios seem like science fiction, consider that Geoffrey Hinton gives a 10–50 percent chance that AI would result in extinction (“X risk”). Yoshua Bengio, the most cited AI researcher and most cited living scientist, gives an X risk of 20 percent. Yudkowski and Yampolskiy give an X risk of over 95 percent, while Dario Amodei, CEO of Anthropic, gives an X risk of 10–25 percent.

Some researchers, such as Yann LeCun (previous chief researcher at Meta), give a near-zero probability of a catastrophic outcome. However, Roman Yampolskiy challenges such confidence. “If you have a solution (to alignment), you can make billions of dollars….not a single AI lab in the world has one. They don’t have a paper published on how to do it. Not a patent, not a rigorous blog post. The best they say is, ‘When we build it, we’ll figure it out.’”

AI risk represents the apex of growth mania, where a few tech elites gamble with humanity’s future. They take these risks without our consent and without our interests in mind. As Anthropic’s Jack Clark opined recently, “…the AI industry has a gas pedal, but it doesn’t have a brake pedal.”

Just like the economy it feeds.

Dave Rollo is a policy specialist and team leader of the Keep Our Counties Great campaign at CASSE and serves on the Bloomington, Indiana City Council. On June 10, Bloomington became the first city to adopt a resolution for a moratorium on Artificial General Intelligence because of its existential risk to humanity.