mlsanityData to 14 Oct 2026

Unsolved Problems in ML Safety

arXiv 2109.13916 ↗First posted 28 Sep 2021cs.LG cs.AI cs.CLcited by 292

Not on the map. Why?

Cited by 32 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Machine learning (ML) systems are rapidly increasing in size, are acquiring new capabilities, and are increasingly deployed in high-stakes settings. As with other powerful technologies, safety for ML should be a leading research priority. In response to emerging safety challenges in ML, such as those introduced by recent large-scale models, we provide a new roadmap for ML Safety and refine the technical problems that the field needs to address. We present four problems ready for research, namely withstanding hazards ("Robustness"), identifying hazards ("Monitoring"), reducing inherent model hazards ("Alignment"), and reducing systemic hazards ("Systemic Safety"). Throughout, we clarify each problem's motivation and provide concrete research directions.

Key facts. Cited by 292 papers in the map (data through 14 Oct 2026). Builds on 119 of its 225 references that are in the map. First posted on arXiv 28 Sep 2021.

Builds on

119 of 225 references.

References · 119

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  7. 2013Intriguing properties of neural networkscited by 5,155
  8. 2017On Calibration of Modern Neural Networkscited by 4,960
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  11. 2021Program Synthesis with Large Language Modelscited by 3,637
  12. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
38 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  2. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  3. 2016Deep Learning with Differential Privacycited by 2,728
  4. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  5. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  6. 2016Equality of Opportunity in Supervised Learningcited by 2,007
  7. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  8. 2016Concrete Problems in AI Safetycited by 1,642
  9. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  10. 2019Natural Adversarial Examplescited by 1,408
  11. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  12. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  13. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  14. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  15. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,106
  16. 2018Deep Anomaly Detection with Outlier Exposurecited by 942
  17. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 901
  18. 2017Ensemble Adversarial Training: Attacks and Defensescited by 840
  19. 2020Aligning AI With Shared Human Valuescited by 723
  20. 2017Evasion Attacks against Machine Learning at Test Timecited by 689
  21. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
  22. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  23. 2020On Adaptive Attacks to Adversarial Example Defensescited by 555
  24. 2015Towards Open Set Deep Networkscited by 538
  25. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  26. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 519
  27. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
  28. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  29. 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 468
  30. 2017Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Modelscited by 462
  31. 2017CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learningcited by 461
  32. 2020Artificial Intelligence, Values and Alignmentcited by 448
  33. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  34. 2020CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instancescited by 441
  35. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
  36. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 422
  37. 2019Unlabeled Data Improves Adversarial Robustnesscited by 416
  38. 2018AI safety via debatecited by 400

50 most cited of 119 All 119 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 39; 2023: 72; 2024: 72; 2025: 58; 2026: 44.

292 papers.

Sort:

By area:

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  2. 2022Emergent Abilities of Large Language Modelscited by 2,512
  3. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  4. 2022Red Teaming Language Models with Language Modelscited by 1,026
  5. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  6. 2021Generalized Out-of-Distribution Detection: A Surveycited by 802
  7. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  9. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  10. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  11. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  12. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
38 more
  1. 2023Tool Learning with Foundation Modelscited by 302
  2. 2022OpenOOD: Benchmarking Generalized Out-of-Distribution Detectioncited by 268
  3. 2022Prompting GPT-3 To Be Reliablecited by 260
  4. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  5. 2023Challenges and Applications of Large Language Modelscited by 254
  6. 2023Large Language Model Alignment: A Surveycited by 243
  7. 2024Generative Representational Instruction Tuningcited by 233
  8. 2023An Overview of Catastrophic AI Riskscited by 222
  9. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 215
  10. 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 198
  11. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 182
  12. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  13. 2023OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detectioncited by 179
  14. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 159
  15. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 156
  16. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 155
  17. 2021A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challengescited by 154
  18. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 151
  19. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 136
  20. 2021PixMix: Dreamlike Pictures Comprehensively Improve Safety Measurescited by 131
  21. 2024Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agentscited by 125
  22. 2023Foundation Models and Fair Usecited by 116
  23. 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 110
  24. 2023How do Language Models Bind Entities in Context?cited by 108
  25. 2024Approaching Human-Level Forecasting with Language Modelscited by 107
  26. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  27. 2023Deception Abilities Emerged in Large Language Modelscited by 101
  28. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 98
  29. 2022When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgmentcited by 93
  30. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 88
  31. 2024DiffuseMix: Label-Preserving Data Augmentation with Diffusion Modelscited by 78
  32. 2024Feedback Loops With Language Models Drive In-Context Reward Hackingcited by 77
  33. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  34. 2025The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systemscited by 72
  35. 2022Forecasting Future World Events with Neural Networkscited by 63
  36. 2022Do Bayesian Neural Networks Need To Be Fully Stochastic?cited by 63
  37. 2023Connecting the Dots in Trustworthy Artificial Intelligence: From AI Principles, Ethics, and Key Requirements to Responsible AI Systems and Regulationcited by 61
  38. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 61

50 most cited of 292 All 292 (JSON)