mlsanityData to 7 Oct 2026

Concrete Problems in AI Safety

arXiv 1606.06565 ↗First posted 21 Jun 2016cs.AI cs.LGcited by 1,582

Not on the map. Why?

Cited by 240 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Rapid progress in machine learning and artificial intelligence (AI) has brought increasing attention to the potential impacts of AI technologies on society. In this paper we discuss one such potential impact: the problem of accidents in machine learning systems, defined as unintended and harmful behavior that may emerge from poor design of real-world AI systems. We present a list of five practical research problems related to accident risk, categorized according to whether the problem originates from having the wrong objective function ("avoiding side effects" and "avoiding reward hacking"), an objective function that is too expensive to evaluate frequently ("scalable supervision"), or undesirable behavior during the learning process ("safe exploration" and "distributional shift"). We review previous work in these areas as well as suggesting research directions with a focus on relevance to cutting-edge AI systems. Finally, we consider the high-level question of how to think most productively about the safety of forward-looking applications of AI.

Key facts. Cited by 1,582 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Jun 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 1; 2018: 8; 2019: 11; 2020: 4; 2021: 166; 2022: 158; 2023: 189; 2024: 233; 2025: 375; 2026: 437.

1,582 papers.

Sort:

By area:

  1. 2017Deep reinforcement learning from human preferencescited by 3,846
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  5. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  7. 2022Red Teaming Language Models with Language Modelscited by 983
  8. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  9. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  10. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  11. 2022A Generalist Agentcited by 802
  12. 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
38 more
  1. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  2. 2021Robust fine-tuning of zero-shot modelscited by 785
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  4. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  5. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  6. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  7. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  8. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  9. 2022Self-critiquing models for assisting human evaluatorscited by 372
  10. 2021True Few-Shot Learning with Language Modelscited by 343
  11. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  12. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  13. 2023Tool Learning with Foundation Modelscited by 302
  14. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  15. 2021Unsolved Problems in ML Safetycited by 291
  16. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  17. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  18. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
  19. 2025Learning to Reason under Off-Policy Guidancecited by 266
  20. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  21. 2022Prompting GPT-3 To Be Reliablecited by 258
  22. 2023Reward Design with Language Modelscited by 257
  23. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  24. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  25. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  26. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  27. 2023Large Language Model Alignment: A Surveycited by 241
  28. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  29. 2023Nash Learning from Human Feedbackcited by 220
  30. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 218
  31. 2022Extremely Simple Activation Shaping for Out-of-Distribution Detectioncited by 212
  32. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  33. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 211
  34. 2021A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detectioncited by 206
  35. 2021Trustworthy AI: From Principles to Practicescited by 206
  36. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  37. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 187
  38. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185

50 most cited of 1,582 All 1,582 (JSON)