mlsanityData to 7 Oct 2026

X-Risk Analysis for AI Research

arXiv 2206.05862 ↗First posted 13 Jun 2022cs.CY cs.AI cs.LGcited by 60

Not on the map. Why?

Abstract (arXiv)

Artificial intelligence (AI) has the potential to greatly improve society, but as with any powerful technology, it comes with heightened risks and responsibilities. Current AI research lacks a systematic discussion of how to manage long-tail risks from AI systems, including speculative long-term risks. Keeping in mind the potential benefits of AI, there is some concern that building ever more intelligent and powerful AI systems could eventually result in systems that are more powerful than us; some say this is like playing with fire and speculate that this could create existential risks (x-risks). To add precision and ground these discussions, we provide a guide for how to analyze AI x-risk, which consists of three parts: First, we review how systems can be made safer today, drawing on time-tested concepts from hazard analysis and systems safety that have been designed to steer large processes in safer directions. Next, we discuss strategies for having long-term impacts on the safety of future systems. Finally, we discuss a crucial concept in making AI systems safer by improving the balance between safety and general capabilities. We hope this document and the presented concepts and tools serve as a useful guide for understanding how to analyze AI x-risk.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Builds on 26 of its 90 references that are in the map. First posted on arXiv 13 Jun 2022.

Builds on

26 of 90 references.

References · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  7. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  8. 2018Deep Anomaly Detection with Outlier Exposurecited by 939
  9. 2020Aligning AI With Shared Human Valuescited by 706
  10. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  11. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 511
  12. 2020Generative Language Modeling for Automated Theorem Provingcited by 379
14 more
  1. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  2. 2021Unsolved Problems in ML Safetycited by 291
  3. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  4. 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 209
  5. 2022Predictability and Surprise in Large Generative Modelscited by 209
  6. 2020Open Problems in Cooperative AIcited by 167
  7. 2018The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigationcited by 159
  8. 2019Optimal Policies Tend to Seek Powercited by 115
  9. 2016The Off-Switch Gamecited by 97
  10. 2020Dialogue Response Ranking Training with Large-Scale Human Feedback Datacited by 68
  11. 2021What Would Jiminy Cricket Do? Towards Agents That Behave Morallycited by 57
  12. 2017When Will AI Exceed Human Performance? Evidence from AI Expertscited by 41
  13. 2020Statistical Consequences of Fat Tails: Real World Preasymptotics, Epistemology, and Applicationscited by 16
  14. 2022Execute Order 66: Targeted Data Poisoning for Reinforcement Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 16; 2024: 18; 2025: 14; 2026: 3.

60 papers.

Sort:

By area:

  1. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  2. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  3. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  4. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  5. 2022OpenOOD: Benchmarking Generalized Out-of-Distribution Detectioncited by 267
  6. 2023Challenges and Applications of Large Language Modelscited by 251
  7. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  8. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  9. 2023An Overview of Catastrophic AI Riskscited by 214
  10. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
  11. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  12. 2022Broken Neural Scaling Lawscited by 102
38 more
  1. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  2. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 89
  3. 2023Can GPT-4 Perform Neural Architecture Search?cited by 74
  4. 2022Forecasting Future World Events with Neural Networkscited by 63
  5. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 56
  6. 2023Evaluating Superhuman Models with Consistency Checkscited by 47
  7. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  8. 2024Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Surveycited by 36
  9. 2023Artificial Influence: An Analysis Of AI-Driven Persuasioncited by 31
  10. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  11. 2025The Traitors: Deception and Trust in Multi-Agent Language Model Simulationscited by 30
  12. 2024Two Types of AI Existential Risk: Decisive and Accumulativecited by 21
  13. 2025The Coming Crisis of Multi-Agent Misalignment: AI Alignment Must Be a Dynamic and Social Processcited by 21
  14. 2022The Expertise Problem: Learning from Specialized Feedbackcited by 20
  15. 2024Holistic Safety and Responsibility Evaluations of Advanced AI Modelscited by 18
  16. 2025Compromising Honesty and Harmlessness in Language Models via Deception Attackscited by 17
  17. 2025LLM-Safety Evaluations Lack Robustnesscited by 16
  18. 2023How Good Are LLMs at Out-of-Distribution Detection?cited by 15
  19. 2023Towards Interpretable Sequence Continuation: Analyzing Shared Circuits in Large Language Modelscited by 14
  20. 2024Unsolvable Problem Detection: Robust Understanding Evaluation for Large Multimodal Modelscited by 12
  21. 2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Driftcited by 12
  22. 2025The Lock-in Hypothesis: Stagnation by Algorithmcited by 11
  23. 2022How Would The Viewer Feel? Estimating Wellbeing From Video Scenarioscited by 9
  24. 2022Robots-Dont-Cry: Understanding Falsely Anthropomorphic Utterances in Dialog Systemscited by 8
  25. 2024MANO: Exploiting Matrix Norm for Unsupervised Accuracy Estimation Under Distribution Shiftscited by 8
  26. 2024Beyond the Safety Bundle: Auditing the Helpful and Harmless Datasetcited by 8
  27. 2025Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectivescited by 8
  28. 2023Hazards from Increasingly Accessible Fine-Tuning of Downloadable Foundation Modelscited by 7
  29. 2023ChatGPT v Bard v Bing v Claude 2 v Aria v human-expert. How good are AI chatbots at scientific writing?cited by 6
  30. 2023Survey on AI Ethics: A Socio-technical Perspectivecited by 4
  31. 2025Psychometric Personality Shaping Modulates Capabilities and Safety in Language Modelscited by 4
  32. 2023Uncovering the Hidden Cost of Model Compressioncited by 3
  33. 2025Unraveling Token Prediction Refinement and Identifying Essential Layers in Language Modelscited by 3
  34. 2025Relevance Isn't All You Need: Scaling RAG Systems With Inference-Time Compute Via Multi-Criteria Rerankingcited by 3
  35. 2025From Directions to Cones: Exploring Multidimensional Representations of Propositional Facts in LLMscited by 3
  36. 2023Fairness in AI and Its Long-Term Implications on Societycited by 2
  37. 2024FlowCon: Out-of-Distribution Detection using Flow-Based Contrastive Learningcited by 2
  38. 2024SAGE: Scalable Ground Truth Evaluations for Large Sparse Autoencoderscited by 2

50 most cited of 60 All 60 (JSON)