mlsanityData to 7 Oct 2026

Adversarial Training for High-Stakes Reliability

arXiv 2205.01663 ↗First posted 3 May 2022cs.LG cs.AI cs.CLcited by 60

Not on the map. Why?

Abstract (arXiv)

In the future, powerful AI systems may be deployed in high-stakes settings, where a single failure could be catastrophic. One technique for improving AI safety in high-stakes settings is adversarial training, which uses an adversary to generate examples to train on in order to achieve better worst-case performance. In this work, we used a safe language generation task (``avoid injuries'') as a testbed for achieving high reliability through adversarial training. We created a series of adversarial training techniques -- including a tool that assists human adversaries -- to find and eliminate failures in a classifier that filters text completions suggested by a generator. In our task, we determined that we can set very conservative classifier thresholds without significantly impacting the quality of the filtered outputs. We found that adversarial training increased robustness to the adversarial attacks that we trained on -- doubling the time for our contractors to find adversarial examples both with our tool (from 13 to 26 minutes) and without (from 20 to 44 minutes) -- without affecting in-distribution performance. We hope to see further work in the high-stakes reliability setting, including more powerful tools for enhancing human adversaries and better ways to measure high levels of reliability, until we can confidently rule out the possibility of catastrophic deployment-time failures of powerful models.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Builds on 52 of its 84 references that are in the map. First posted on arXiv 3 May 2022.

Builds on

52 of 84 references.

References · 52

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
38 more
  1. 2016Concrete Problems in AI Safetycited by 1,582
  2. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  3. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  4. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  5. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  7. 2022Red Teaming Language Models with Language Modelscited by 983
  8. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  9. 2017Constrained Policy Optimizationcited by 928
  10. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  11. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  12. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  13. 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 564
  14. 2017A systematic study of the class imbalance problem in convolutional neural networkscited by 510
  15. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  16. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  17. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  18. 2018AI safety via debatecited by 384
  19. 2021Unsolved Problems in ML Safetycited by 291
  20. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  21. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  22. 2021Mastering Atari Games with Limited Datacited by 235
  23. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  24. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  25. 2018Adversarial Risk and the Dangers of Evaluating Against Weak Attackscited by 221
  26. 2018Semidefinite relaxations for certifying robustness to adversarial examplescited by 178
  27. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  28. 2021Improving Question Answering Model Robustness with Synthetic Adversarial Data Generationcited by 77
  29. 2021Tailor: Generating and Perturbing Text with Semantic Controlscited by 67
  30. 2021Improved OOD Generalization via Adversarial Training and Pre-trainingcited by 66
  31. 2018On the Suitability of Lp-norms for Creating and Preventing Adversarial Examplescited by 57
  32. 2018A Statistical Approach to Assessing Neural Network Robustnesscited by 54
  33. 2020Interactive Weak Supervision: Learning Useful Heuristics for Data Labelingcited by 52
  34. 2018Unrestricted Adversarial Examplescited by 40
  35. 2021Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistantscited by 31
  36. 2020Bayesian Robust Optimization for Imitation Learningcited by 28
  37. 2021Robust Feature-Level Adversaries are Interpretability Toolscited by 27
  38. 2022MuZero with Self-competition for Rate Control in VP9 Video Compressioncited by 27

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 23; 2024: 17; 2025: 9; 2026: 5.

60 papers.

Sort:

By area:

  1. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  4. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  5. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  6. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  7. 2023Pretraining Language Models with Human Preferencescited by 245
  8. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 207
  9. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  10. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  11. 2023Model evaluation for extreme riskscited by 167
  12. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
38 more
  1. 2023Auditing large language models: a three-layered approachcited by 135
  2. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  3. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  4. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  5. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  6. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 107
  7. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
  8. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  9. 2024Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Modelscited by 42
  10. 2023Circuit Breaking: Removing Model Behaviors with Targeted Ablationcited by 27
  11. 2024Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learningcited by 23
  12. 2023Red Teaming Deep Neural Networks with Feature Synthesis Toolscited by 21
  13. 2023Towards Evaluating AI Systems for Moral Status Using Self-Reportscited by 21
  14. 2024Endless Jailbreaks with Bijection Learningcited by 18
  15. 2023PrivacyMind: Large Language Models Can Be Contextual Privacy Protection Learnerscited by 13
  16. 2023ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Modelscited by 12
  17. 2023ALMANACS: A Simulatability Benchmark for Language Model Explainabilitycited by 11
  18. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Modelscited by 11
  19. 2025ROC-n-reroll: How verifier imperfection affects test-time scalingcited by 11
  20. 2023A Holistic Assessment of the Reliability of Machine Learning Systemscited by 9
  21. 2023In Search of the Long-Tail: Systematic Generation of Long-Tail Inferential Knowledge via Logical Rule Guided Searchcited by 9
  22. 2025The Singapore Consensus on Global AI Safety Research Prioritiescited by 9
  23. 2023Fast Propagation is Better: Accelerating Single-Step Adversarial Training via Sampling Subnetworkscited by 8
  24. 2024The Role of Learning Algorithms in Collective Actioncited by 8
  25. 2023Adversarial Fine-Tuning of Language Models: An Iterative Optimisation Approach for the Generation and Detection of Problematic Contentcited by 7
  26. 2024JailbreakLens: Visual Analysis of Jailbreak Attacks Against Large Language Modelscited by 7
  27. 2022Diagnostics for Deep Neural Networks with Automated Copy/Paste Attackscited by 6
  28. 2024Adversarial Training: A Surveycited by 6
  29. 2025Bypassing Prompt Guards in Production with Controlled-Release Promptingcited by 6
  30. 2023Can Large Language Models Change User Preference Adversarially?cited by 5
  31. 2023Intent-aligned AI systems deplete human agency: the need for agency foundations research in AI safetycited by 4
  32. 2025Misalignment or misuse? The AGI alignment tradeoffcited by 4
  33. 2023Alignment is not sufficient to prevent large language models from generating harmful information: A psychoanalytic perspectivecited by 3
  34. 2024Dynamic Normativity: Necessary and Sufficient Conditions for Value Alignmentcited by 3
  35. 2023Large Language Models Sometimes Generate Purely Negatively-Reinforced Textcited by 2
  36. 2026Safe RLHF Beyond Expectation: Stochastic Dominance for Universal Spectral Risk Controlcited by 2
  37. 2024Aligning language models with human preferencescited by 1
  38. 2025Geometry-Guided Adversarial Prompt Detection via Curvature and Local Intrinsic Dimensioncited by 1

50 most cited of 60 All 60 (JSON)