mlsanityData to 14 Oct 2026

An Overview of Catastrophic AI Risks

arXiv 2306.12001 ↗First posted 21 Jun 2023cs.CY cs.AI cs.LGcited by 222

Not on the map. Why?

Cited by 33 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Rapid advancements in artificial intelligence (AI) have sparked growing concerns among experts, policymakers, and world leaders regarding the potential for increasingly advanced AI systems to pose catastrophic risks. Although numerous risks have been detailed separately, there is a pressing need for a systematic discussion and illustration of the potential dangers to better inform efforts to mitigate them. This paper provides an overview of the main sources of catastrophic AI risks, which we organize into four categories: malicious use, in which individuals or groups intentionally use AIs to cause harm; AI race, in which competitive environments compel actors to deploy unsafe AIs or cede control to AIs; organizational risks, highlighting how human factors and complex systems can increase the chances of catastrophic accidents; and rogue AIs, describing the inherent difficulty in controlling agents far more intelligent than humans. For each category of risk, we describe specific hazards, present illustrative stories, envision ideal scenarios, and propose practical suggestions for mitigating these dangers. Our goal is to foster a comprehensive understanding of these risks and inspire collective and proactive efforts to ensure that AIs are developed and deployed in a safe manner. Ultimately, we hope this will allow us to realize the benefits of this powerful technology while minimizing the potential for catastrophic outcomes.

Key facts. Cited by 222 papers in the map (data through 14 Oct 2026). Builds on 31 of its 149 references that are in the map. First posted on arXiv 21 Jun 2023.

Builds on

31 of 149 references.

References · 31

  1. 2013Intriguing properties of neural networkscited by 5,155
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2018Datasheets for Datasetscited by 1,386
  7. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  8. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  9. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
  10. 2022In-context Learning and Induction Headscited by 964
  11. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  12. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
19 more
  1. 2019Release Strategies and the Social Impacts of Language Modelscited by 513
  2. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  3. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 313
  4. 2021Unsolved Problems in ML Safetycited by 292
  5. 2023LEACE: Perfect linear concept erasure in closed formcited by 267
  6. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 182
  7. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Modelscited by 110
  8. 2016The Off-Switch Gamecited by 97
  9. 2020Trojaning Language Models for Fun and Profitcited by 90
  10. 2017Online Human-Bot Interactions: Detection, Estimation, and Characterizationcited by 64
  11. 2022X-Risk Analysis for AI Researchcited by 60
  12. 2021The Threat of Offensive AI to Organizationscited by 38
  13. 2023Can large language models democratize access to dual-use biotechnology?cited by 35
  14. 2023Artificial Influence: An Analysis Of AI-Driven Persuasioncited by 33
  15. 2021What are you optimizing for? Aligning Recommender Systems with Human Valuescited by 29
  16. 2023Towards best practices in AGI safety and governance: A survey of expert opinioncited by 27
  17. 2023Natural Selection Favors AIs over Humanscited by 24
  18. 2023What does it take to catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training via Compute Monitoringcited by 21
  19. 2018Robust Artificial Intelligence and Robust Human Organizationscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 26; 2024: 73; 2025: 73; 2026: 50.

222 papers.

Sort:

By area:

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  2. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  3. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  4. 2023Language Models Represent Space and Timecited by 316
  5. 2024Rethinking Machine Unlearning for Large Language Modelscited by 257
  6. 2023Large Language Model Alignment: A Surveycited by 243
  7. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 223
  8. 2023Can LLM-Generated Misinformation Be Detected?cited by 177
  9. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  10. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 146
  11. 2023Taken out of context: On measuring situational awareness in LLMscited by 143
  12. 2024Language Models Learn to Mislead Humans via RLHFcited by 124
38 more
  1. 2024Wav-KAN: Wavelet Kolmogorov-Arnold Networkscited by 122
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  3. 2024Universal Neurons in GPT2 Language Modelscited by 102
  4. 2023Deception Abilities Emerged in Large Language Modelscited by 101
  5. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 98
  6. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 96
  7. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 93
  8. 2024AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policiescited by 74
  9. 2025SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learningcited by 72
  10. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 71
  11. 2025SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agentscited by 71
  12. 2025Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIscited by 67
  13. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 65
  14. 2024Nicer Than Humans: How do Large Language Models Behave in the Prisoner's Dilemma?cited by 63
  15. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 62
  16. 2024Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMscited by 59
  17. 2024Large Language Model Safety: A Holistic Surveycited by 59
  18. 2024LlavaGuard: An Open VLM-based Framework for Safeguarding Vision Datasets and Modelscited by 52
  19. 2024Sabotage Evaluations for Frontier Modelscited by 52
  20. 2024Virtual Personas for Language Models via an Anthology of Backstoriescited by 51
  21. 2023Managing extreme AI risks amid rapid progresscited by 48
  22. 2024Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablationcited by 48
  23. 2023The Persuasive Power of Large Language Modelscited by 47
  24. 2024A Safe Harbor for AI Evaluation and Red Teamingcited by 47
  25. 2023Testing Language Model Agents Safely in the Wildcited by 46
  26. 2024Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Modelscited by 44
  27. 2023Honesty Is the Best Policy: Defining and Mitigating AI Deceptioncited by 40
  28. 2023Interpretable Diffusion via Information Decompositioncited by 37
  29. 2024Personality Alignment of Large Language Modelscited by 36
  30. 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
  31. 2024Towards evaluations-based safety cases for AI schemingcited by 33
  32. 2024Adversaries Can Misuse Combinations of Safe Modelscited by 32
  33. 2024Persuasion with Large Language Models: A Survey of Empirical Evidence, Study Methodologies, and Ethical Implicationscited by 32
  34. 2024AI-LieDar: Examine the Trade-off Between Utility and Truthfulness in LLM Agentscited by 31
  35. 2025Representation Bending for Large Language Model Safetycited by 30
  36. 2026The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systemscited by 30
  37. 2023Provably safe systems: the only path to controllable AGIcited by 29
  38. 2024Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoderscited by 27

50 most cited of 222 All 222 (JSON)