# Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path? arXiv: https://arxiv.org/abs/2502.15657 ยท HTML: https://mlsanity.com/paper/2502.15657/ ## Key facts - Cited by 75 papers in the map (data through 14 Oct 2026). - Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. - Builds on 0 of its 4 references that are in the map. - First posted on arXiv 21 Feb 2025. ## Placed in - LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/) ## Built on by (most cited first, 30 of 75) - [The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search](https://mlsanity.com/paper/2504.08066/) (2025) - [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025) - [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025) - [Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents](https://mlsanity.com/paper/2503.24047/) (2025) - [From Automation to Autonomy: A Survey on Large Language Models in Scientific Discovery](https://mlsanity.com/paper/2505.13259/) (2025) - [From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery](https://mlsanity.com/paper/2508.14111/) (2025) - [Security Concerns for Large Language Models: A Survey](https://mlsanity.com/paper/2505.18889/) (2025) - [A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents](https://mlsanity.com/paper/2506.23844/) (2025) - [The Coming Crisis of Multi-Agent Misalignment: AI Alignment Must Be a Dynamic and Social Process](https://mlsanity.com/paper/2506.01080/) (2025) - [AI Scientists Fail Without Strong Implementation Capability](https://mlsanity.com/paper/2506.01372/) (2025) - [Assessing Adaptive World Models in Machines with Novel Games](https://mlsanity.com/paper/2507.12821/) (2025) - [General agents contain world models](https://mlsanity.com/paper/2506.01622/) (2025) - [SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents](https://mlsanity.com/paper/2505.23559/) (2025) - [Offline Model-Based Optimization: Comprehensive Review](https://mlsanity.com/paper/2503.17286/) (2025) - [How Far Are AI Scientists from Changing the World?](https://mlsanity.com/paper/2507.23276/) (2025) - [Avoiding Obfuscation with Prover-Estimator Debate](https://mlsanity.com/paper/2506.13609/) (2025) - [When Benign Inputs Lead to Severe Harms: Eliciting Unsafe Unintended Behaviors of Computer-Use Agents](https://mlsanity.com/paper/2602.08235/) (2026) - [On the Impossibility of Separating Intelligence from Judgment: The Computational Intractability of Filtering for AI Alignment](https://mlsanity.com/paper/2507.07341/) (2025) - [The Singapore Consensus on Global AI Safety Research Priorities](https://mlsanity.com/paper/2506.20702/) (2025) - [OpenFake: An Open Dataset and Platform Toward Real-World Deepfake Detection](https://mlsanity.com/paper/2509.09495/) (2025) - [Build Agent Advocates, Not Platform Agents](https://mlsanity.com/paper/2505.04345/) (2025) - [AI Awareness](https://mlsanity.com/paper/2504.20084/) (2025) - [How are AI agents used? Evidence from 177,000 MCP tools](https://mlsanity.com/paper/2603.23802/) (2026) - [Propensity Inference: Environmental Contributors to LLM Behaviour](https://mlsanity.com/paper/2604.21098/) (2026) - [Bench-2-CoP: Can We Trust Benchmarking for EU AI Compliance?](https://mlsanity.com/paper/2508.05464/) (2025) - [Toward a Safe Internet of Agents](https://mlsanity.com/paper/2512.00520/) (2025) - [The Other Mind: How Language Models Exhibit Human Temporal Cognition](https://mlsanity.com/paper/2507.15851/) (2025) - [Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training](https://mlsanity.com/paper/2605.11134/) (2026) - [Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security](https://mlsanity.com/paper/2605.23989/) (2026) - [Military AI Needs Technically-Informed Regulation to Safeguard AI Research and its Applications](https://mlsanity.com/paper/2505.18371/) (2025) ## Abstract (arXiv) The leading AI companies are increasingly focused on building generalist AI agents -- systems that can autonomously plan, act, and pursue goals across almost all tasks that humans can perform. Despite how useful these systems might be, unchecked AI agency poses significant risks to public safety and security, ranging from misuse by malicious actors to a potentially irreversible loss of human control. We discuss how these risks arise from current AI training methods. Indeed, various scenarios and experiments have demonstrated the possibility of AI agents engaging in deception or pursuing goals that were not specified by human operators and that conflict with human interests, such as self-preservation. Following the precautionary principle, we see a strong need for safer, yet still useful, alternatives to the current agency-driven trajectory. Accordingly, we propose as a core building block for further advances the development of a non-agentic AI system that is trustworthy and safe by design, which we call Scientist AI. This system is designed to explain the world from observations, as opposed to taking actions in it to imitate or please humans. It comprises a world model that generates theories to explain data and a question-answering inference machine. Both components operate with an explicit notion of uncertainty to mitigate the risks of overconfident predictions. In light of these considerations, a Scientist AI could be used to assist human researchers in accelerating scientific progress, including in AI safety. In particular, our system can be employed as a guardrail against AI agents that might be created despite the risks involved. Ultimately, focusing on non-agentic AI may enable the benefits of AI innovation while avoiding the risks associated with the current trajectory. We hope these arguments will motivate researchers, developers, and policymakers to favor this safer path.