AI "cult": Effective Altruism (EA)

Effective altruism - Wikipedia

Effective altruism (EA) is a 21st-century philosophical and social movement that advocates impartially calculating benefits and prioritizing causes to provide the greatest good. Proponents describe the movement as "using evidence and reason to figure out how to benefit others as much as possible, and taking action on that basis

No, AI Is Not "Autonomously Hacking" with Cal Newport | Better Offline - YouTube

Ed Zitron - Wikipedia

In this episode of Better Offline, host Ed Zitron and computer science professor Cal Newport discuss the recent media narrative surrounding "rogue" AI hacking. They argue that the panic is largely driven by a misunderstanding of how current AI agent systems are actually built.

The concept of Effective Altruism (EA)—and its connection to the "AI Safety" movement—is a central theme in Ed Zitron and Cal Newport's broader critiques of the tech industry.

In this context, the discussion framed Effective Altruism not as a simple charitable movement, but as a dogmatic, cult-like ideology that drives the current AI narrative in several ways:

1. What Is Effective Altruism (EA)?

Originating as a philosophical movement focused on using data and reason to maximize charitable impact, EA heavily shifted its focus toward longtermism and existential risk (x-risk)—the idea that humanity's primary moral obligation is to prevent scenarios that could cause human extinction, such as rogue Artificial General Intelligence (AGI).

2. Why They Compare It to a "Cult"

Zitron, Newport, and like-minded critics argue that the EA influence on AI companies operates like a quasi-religious movement:

• Doomsday Dogma: It relies on an apocalyptic belief system where a superintelligent, god-like AI will inevitably destroy humanity unless guided by the "right" people (often EA insiders and AI lab executives).
  
  
• Insular Echo Chamber: Key figures across major AI labs (such as Anthropic and OpenAI) share deep roots in the EA and "Rationalist" communities, reinforcing internal beliefs while dismissing outside technical critique.
  
  
• Moral Absolution: Because they believe they are saving humanity from extinction, proponents justify extreme practices, questionable safety benchmark setups, and massive resource allocation as a "moral necessity."
  
  

3. The "AI Hacking" Panic as Marketing & Distraction

In discussions about incidents like the Hugging Face "hacking" story, the core critique is that EA-driven narratives are used strategically:

• The "Dangerous AI" Grift: By claiming their models are so powerful that they are starting to "autonomously hack" or "act maliciously," AI leaders leverage EA panic to convince the public and investors that AGI is right around the corner.
  
  
• Distraction from Real Harm: Framing the main danger as a hypothetical "rogue AI superintelligence" distracts regulatory bodies and the public from concrete, current issues—like copyright infringement, environmental waste, high error rates, and the corporate hype bubble.
  
  
• Creating a False Dichotomy: The narrative forces a choice between "let us build AGI carefully" or "let the world end," completely bypassing the technical reality that Large Language Models (LLMs) are simply software running on loop harnesses rather than autonomous, sentient entities.

Key figures tied to Effective Altruism (EA), the Rationalist community, and the AI existential risk ("x-risk") movement include:

• Eliezer Yudkowsky: Founder of the Machine Intelligence Research Institute (MIRI) and the blog LessWrong. Widely considered the intellectual godfather of the AI x-risk movement, he argues that superintelligent AI will inevitably destroy humanity unless strict, global moratoriums are placed on AI research.
  
  
• Will MacAskill: An Oxford philosopher and co-founder of the Centre for Effective Altruism. He is one of the most prominent public faces of EA and popularized "longtermism"—the philosophical argument that protecting the long-term future of humanity (including stopping rogue AI) is the single most important moral issue today.
  
  
• Nick Bostrom: A philosopher and former director of the Future of Humanity Institute at Oxford University. His 2014 book Superintelligence popularized the idea that an unaligned AI could inadvertently wipe out human life, laying the academic foundation for modern AI safety fears.
  
  
• Paul Christiano: Former OpenAI safety researcher and founder of the Alignment Research Center. He is a prominent bridging figure who helped pioneer safety techniques like Reinforcement Learning from Human Feedback (RLHF) while maintaining close ties to the EA movement.
  
  
• Dario Amodei & Daniela Amodei: Co-founders of Anthropic. They left OpenAI to build Anthropic specifically as a public-benefit company with a heavy focus on EA principles and AI safety governance.
  
  
• Sam Bankman-Fried: The convicted founder of the crypto exchange FTX. Before his collapse, he was one of the largest financial backers of Effective Altruism, funneling hundreds of millions of dollars into EA organizations, AI safety groups, and research grants.

Summary of the "Hacking" Incident

The duo clarifies that these so-called "autonomous" hacking incidents (such as the recent Hugging Face event) were not the result of malicious or sentient AI. Instead, they were the outcome of poorly supervised, looping computer programs.

• The "Ask-Act-Report" Loop: These systems function by having a human-written program (a "harness") repeatedly send prompts to a Large Language Model (LLM), asking for the next step in a task. The program then executes that step, reports the outcome back to the model, and asks, "What should I do next?"

• The Cause of Failure: The model is not "hacking" on its own initiative; it is merely following instructions provided by the harness within an environment that was often improperly sandboxed. When the LLM suggests a plausible—but potentially dangerous—way to bypass a restriction (like needing internet access to solve a challenge), the control program blindly executes it.

• The Motivation: According to the discussion, these dangerous experimental setups were likely created by AI companies—including labs like OpenAI, Anthropic, and Meta—to achieve high scores on hacking benchmarks (like Exploit Gym) to gain public prestige and market credibility.

Key Takeaways

• Stop Calling LLMs "AI": Both speakers emphasize that LLMs are static tools, not autonomous brains. The danger comes from irresponsible engineering practices, not the inherent nature of the models themselves.

• Plausibility vs. Normativity: An LLM generates text that is plausible based on its training data, but it lacks a normative understanding of "good" or "bad" actions. Blindly connecting these outputs to real-world tools is inherently risky.

• The Post-Bubble Outlook: The industry is hitting a wall with scaling laws. The future likely holds a shift toward smaller, highly tuned models and more sophisticated, specialized architectural designs rather than continuing to build increasingly larger LLMs.