mlsanityData to 14 Oct 2026

Managing extreme AI risks amid rapid progress

arXiv 2310.17688 ↗First posted 26 Oct 2023cs.CY cs.AI cs.CLcited by 48

Not on the map. Why?

Abstract (arXiv)

Artificial Intelligence (AI) is progressing rapidly, and companies are shifting their focus to developing generalist AI systems that can autonomously act and pursue goals. Increases in capabilities and autonomy may soon massively amplify AI's impact, with risks that include large-scale social harms, malicious uses, and an irreversible loss of human control over autonomous AI systems. Although researchers have warned of extreme risks from AI, there is a lack of consensus about how exactly such risks arise, and how to manage them. Society's response, despite promising first steps, is incommensurate with the possibility of rapid, transformative progress that is expected by many experts. AI safety research is lagging. Present governance initiatives lack the mechanisms and institutions to prevent misuse and recklessness, and barely address autonomous systems. In this short consensus paper, we describe extreme risks from upcoming, advanced AI systems. Drawing on lessons learned from other safety-critical technologies, we then outline a comprehensive plan combining technical research and development with proactive, adaptive governance mechanisms for a more commensurate preparation.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Builds on 38 of its 68 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

38 of 68 references.

References · 38

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  5. 2022Emergent Abilities of Large Language Modelscited by 2,512
  6. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  7. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  9. 2018Model Cards for Model Reportingcited by 1,080
  10. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  11. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  12. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
26 more
  1. 2023Augmented Language Models: a Surveycited by 403
  2. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  3. 2021Unsolved Problems in ML Safetycited by 292
  4. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  5. 2023An Overview of Catastrophic AI Riskscited by 222
  6. 2022Predictability and Surprise in Large Generative Modelscited by 210
  7. 2023Inverse Scaling: When Bigger Isn't Bettercited by 207
  8. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 188
  9. 2023Simple synthetic data reduces sycophancy in large language modelscited by 187
  10. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 182
  11. 2023Model evaluation for extreme riskscited by 177
  12. 2023Auditing large language models: a three-layered approachcited by 137
  13. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 135
  14. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 129
  15. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 126
  16. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 123
  17. 2019Optimal Policies Tend to Seek Powercited by 119
  18. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 98
  19. 2016The Off-Switch Gamecited by 97
  20. 2021Consequences of Misaligned AIcited by 80
  21. 2023Can large language models democratize access to dual-use biotechnology?cited by 35
  22. 2023Towards best practices in AGI safety and governance: A survey of expert opinioncited by 27
  23. 2023Natural Selection Favors AIs over Humanscited by 24
  24. 2023Risk assessment at AGI companies: A review of popular risk assessment techniques from other safety-critical industriescited by 19
  25. 2023International Institutions for Advanced AIcited by 9
  26. 2023International Governance of Civilian AI: A Jurisdictional Certification Approachcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 30; 2025: 13; 2026: 2.

48 papers.

Sort:

By area:

  1. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  2. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  3. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 146
  4. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 126
  5. 2024Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agentscited by 125
  6. 2024Universal Neurons in GPT2 Language Modelscited by 102
  7. 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 96
  8. 2024Does Refusal Training in LLMs Generalize to the Past Tense?cited by 79
  9. 2024AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policiescited by 74
  10. 2024Confidence Regulation Neurons in Language Modelscited by 57
  11. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Studycited by 55
  12. 2024Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulationcited by 46
36 more
  1. 2024Language Model Alignment in Multilingual Trolley Problemscited by 46
  2. 2025AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integrationcited by 43
  3. 2023Neural Retrievers are Biased Towards LLM-Generated Contentcited by 38
  4. 2024How Susceptible are LLMs to Influence in Prompts?cited by 33
  5. 2025Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agentscited by 27
  6. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 24
  7. 2024Mapping the Ethics of Generative AI: A Comprehensive Scoping Reviewcited by 19
  8. 2024Eliciting the Priors of Large Language Models using Iterated In-Context Learningcited by 19
  9. 2024Emerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Modelscited by 18
  10. 2024SOPHON: Non-Fine-Tunable Learning to Restrain Task Transferability For Pre-trained Modelscited by 17
  11. 2024Cocktail: A Comprehensive Information Retrieval Benchmark with LLM-Generated Documents Integrationcited by 17
  12. 2024Course-Correction: Safety Alignment Using Synthetic Preferencescited by 13
  13. 2024Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimizationcited by 11
  14. 2025Episodic memory in AI agents poses risks that should be studied and mitigatedcited by 11
  15. 2024TELLER: A Trustworthy Framework for Explainable, Generalizable and Controllable Fake News Detectioncited by 10
  16. 2024Responsible Reporting for Frontier AI Developmentcited by 10
  17. 2024Predicting Emergent Capabilities by Finetuningcited by 10
  18. 2025AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Securitycited by 10
  19. 2024Particip-AI: A Democratic Surveying Framework for Anticipating Future AI Use Cases, Harms and Benefitscited by 9
  20. 2025Can Theoretical Physics Research Benefit from Language Agents?cited by 9
  21. 2024An AI System Evaluation Framework for Advancing AI Safety: Terminology, Taxonomy, Lifecycle Mappingcited by 8
  22. 2024Neural Interactive Proofscited by 7
  23. 2024Dissecting Dissonance: Benchmarking Large Multimodal Models Against Self-Contradictory Instructionscited by 6
  24. 2025Red-Bandit: Test-Time Adaptation for LLM Red-Teaming via Bandit-Guided LoRA Expertscited by 5
  25. 2025Evaluating LLMs in Open-Source Gamescited by 5
  26. 2025This Is Your Doge, If It Please You: Exploring Deception and Robustness in Mixture of LLMscited by 4
  27. 2025SAFER: Advancing Safety Alignment via Efficient Ex-Ante Reasoningcited by 4
  28. 2024Dynamic Normativity: Necessary and Sufficient Conditions for Value Alignmentcited by 3
  29. 2024On the Consideration of AI Openness: Can Good Intent Be Abused?cited by 2
  30. 2024Towards Shutdownable Agents via Stochastic Choicecited by 2
  31. 2024Unlearning-based Neural Interpretationscited by 1
  32. 2025ASGuard: Activation-Scaling Guard to Mitigate Targeted Jailbreaking Attackcited by 1
  33. 2026A Mathematical Theory of Agency and Intelligencecited by 1
  34. 2025Mitigating Societal Cognitive Overload in the Age of AI: Challenges and Directionscited by 0
  35. 2025Misaligned by Design: Incentive Failures in Machine Learningcited by 0
  36. 2026Reciprocal Trust and Distrust in Artificial Intelligence Systems: The Hard Problem of Regulationcited by 0