mlsanityData to 14 Oct 2026

Position: Adversarial ML for LLMs Is Not Making Any Progress

arXiv 2502.02260 ↗First posted 4 Feb 2025cs.LG cs.CRcited by 15

Not on the map. Why?

Abstract (arXiv)

In the past decade, considerable research effort has been devoted to securing machine learning (ML) models that operate in adversarial settings. Yet, progress has been slow even for simple "toy" problems (e.g., robustness to small adversarial perturbations) and is often hindered by non-rigorous evaluations. Today, adversarial ML research has shifted towards studying larger, general-purpose language models. In this position paper, we argue that the situation is now even worse: in the era of LLMs, the field of adversarial ML studies problems that are (1) less clearly defined, (2) harder to solve, and (3) even more challenging to evaluate. As a result, we caution that yet another decade of work on adversarial ML may be failing to produce meaningful progress.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 76 of its 101 references that are in the map. First posted on arXiv 4 Feb 2025.

Builds on

76 of 101 references.

References · 76

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  6. 2013Intriguing properties of neural networkscited by 5,155
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  11. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
38 more
  1. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  2. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  5. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  6. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  8. 2019Machine Unlearningcited by 1,061
  9. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  10. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 920
  11. 2021Membership Inference Attacks From First Principlescited by 878
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  13. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  14. 2017Evasion Attacks against Machine Learning at Test Timecited by 689
  15. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  16. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  17. 2020On Adaptive Attacks to Adversarial Example Defensescited by 555
  18. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  19. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  20. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  21. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  22. 2019On Evaluating Adversarial Robustnesscited by 447
  23. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  24. 2024A StrongREJECT for Empty Jailbreakscited by 385
  25. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 381
  26. 2023Are aligned neural networks adversarially aligned?cited by 315
  27. 2023Poisoning Language Models During Instruction Tuningcited by 269
  28. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Modelscited by 261
  29. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 244
  30. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  31. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 214
  32. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
  33. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 195
  34. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 181
  35. 2020Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defensescited by 180
  36. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
  37. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 163
  38. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 158

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 4.

15 papers.

Sort:

By area:

  1. 2025When Your Reviewer is an LLM: Biases, Divergence, and Prompt Injection Risks in Peer Reviewcited by 19
  2. 2026Shaping capabilities with token-level data filteringcited by 13
  3. 2025Adversarial Tokenizationcited by 11
  4. 2025Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectivescited by 8
  5. 2025Benchmarking Misuse Mitigation Against Covert Adversariescited by 8
  6. 2025Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrievalcited by 7
  7. 2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Researchcited by 6
  8. 2025One Token Embedding Is Enough to Deadlock Your Large Reasoning Modelcited by 5
  9. 2026Memory poisoning and secure multi-agent systemscited by 4
  10. 2026Intent Laundering: AI Safety Datasets Are Not What They Seemcited by 3
  11. 2025A Red Teaming Roadmap Towards System-Level Safetycited by 2
  12. 2025Multi-Agent Framework for Threat Mitigation and Resilience in AI-Based Systemscited by 2
3 more
  1. 2025`A la recherche du sens perdu: your favourite LLM might have more to say than you can understandcited by 0
  2. 2025Keep It Real: Challenges in Attacking Compression-Based Adversarial Purificationcited by 0
  3. 2026An Investigation of the NeurIPS and ICML 2025 Position Trackscited by 0