mlsanityData to 7 Oct 2026

Adversarial Attacks and Defenses in Large Language Models: Old and New Threats

arXiv 2310.19737 ↗First posted 30 Oct 2023cs.AIcited by 49

Not on the map. Why?

Cited by 26 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Over the past decade, there has been extensive research aimed at enhancing the robustness of neural networks, yet this problem remains vastly unsolved. Here, one major impediment has been the overestimation of the robustness of new defense approaches due to faulty defense evaluations. Flawed robustness evaluations necessitate rectifications in subsequent works, dangerously slowing down the research and providing a false sense of security. In this context, we will face substantial challenges associated with an impending adversarial arms race in natural language processing, specifically with closed-source Large Language Models (LLMs), such as ChatGPT, Google Bard, or Anthropic's Claude. We provide a first set of prerequisites to improve the robustness assessment of new approaches and reduce the amount of faulty evaluations. Additionally, we identify embedding space attacks on LLMs as another viable threat model for the purposes of generating malicious content in open-sourced models. Finally, we demonstrate on a recently proposed defense that, without LLM-specific best practices in place, it is easy to overestimate the robustness of a new approach.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 26 of its 28 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

26 of 28 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  9. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  10. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  11. 2019Natural Adversarial Examplescited by 1,401
  12. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
14 more
  1. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  2. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  3. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  4. 2020On Adaptive Attacks to Adversarial Example Defensescited by 547
  5. 2019On Evaluating Adversarial Robustnesscited by 442
  6. 2023Are aligned neural networks adversarially aligned?cited by 305
  7. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  8. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  9. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  10. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 128
  11. 2023Are Defenses for Graph Neural Networks Robust?cited by 38
  12. 2021Exploring Misclassifications of Robust Neural Networks to Enhance Adversarial Attackscited by 33
  13. 2023Natural Selection Favors AIs over Humanscited by 24
  14. 2022Improving Robustness against Real-World and Worst-Case Distribution Shifts through Decision Region Quantificationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 20; 2025: 19; 2026: 9.

49 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  2. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  3. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  4. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  5. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  6. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  7. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Modelscited by 95
  8. 2024Attacking Large Language Models with Projected Gradient Descentcited by 91
  9. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 88
  10. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  11. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 60
  12. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
37 more
  1. 2024Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Modelscited by 23
  2. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 23
  3. 2024Towards Resilient and Efficient LLMs: A Comparative Study of Efficiency, Performance, and Adversarial Robustnesscited by 22
  4. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Modelscited by 21
  5. 2026A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustnesscited by 20
  6. 2024Adversarial Attacks on Large Language Models Using Regularized Relaxationcited by 12
  7. 2024JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMscited by 10
  8. 2024Characterizing and Evaluating the Reliability of LLMs against Jailbreak Attackscited by 8
  9. 2025Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectivescited by 8
  10. 2025LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasurescited by 8
  11. 2025Beyond Data Filtering: Knowledge Localization for Capability Removal in LLMscited by 8
  12. 2025Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMscited by 7
  13. 2025Same Question, Different Words: A Latent Adversarial Framework for Prompt Robustnesscited by 5
  14. 2025Safe-SAIL: Towards a Fine-grained Safety Landscape of Large Language Models via Sparse Autoencoder Interpretation Frameworkcited by 5
  15. 2025Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questionscited by 4
  16. 2025Fast Proxies for LLM Robustness Evaluationcited by 3
  17. 2025Misaligned Roles, Misplaced Images: Structural Input Perturbations Expose Multimodal Alignment Blind Spotscited by 3
  18. 2024Large-Scale Dataset Pruning in Adversarial Training through Data Importance Extrapolationcited by 2
  19. 2024Continuous Embedding Attacks via Clipped Inputs in Jailbreaking Large Language Modelscited by 2
  20. 2024Optimizing AI-Assisted Code Generationcited by 2
  21. 2025Circumventing Safety Alignment in Large Language Models Through Embedding Space Toxicity Attenuationcited by 2
  22. 2025Securing Educational LLMs: A Generalised Taxonomy of Attacks on LLMs and DREAD Risk Assessmentcited by 2
  23. 2026When AI reviews science: Can we trust the referee?cited by 2
  24. 2025Geometry-Guided Adversarial Prompt Detection via Curvature and Local Intrinsic Dimensioncited by 1
  25. 2025Can LLMs Deceive CLIP? Benchmarking Adversarial Compositionality of Pre-trained Multimodal Representation via Text Updatescited by 1
  26. 2025Attacks and Defenses Against LLM Fingerprintingcited by 1
  27. 2025Red Teaming Multimodal Language Models: Evaluating Harm Across Prompt Modalities and Modelscited by 1
  28. 2026Needle-in-RAG: Prompt-Conditioned Character-Level Traceback of Poisoned Spans in Retrieved Evidencecited by 1
  29. 2024On Adversarial Robustness of Language Models in Transfer Learningcited by 0
  30. 2025Proactive Disentangled Modeling of Trigger-Object Pairings for Backdoor Defensecited by 0
  31. 2025Embedding Poisoning: Bypassing Safety Alignment via Embedding Semantic Shiftcited by 0
  32. 2026Sockpuppetting: Jailbreaking LLMs by Combining Prefilling with Optimizationcited by 0
  33. 2026Alignment Drift in Multimodal LLMs: A Two-Phase, Longitudinal Evaluation of Harm Across Eight Model Releasescited by 0
  34. 2026Closing the Distribution Gap in Adversarial Training for LLMscited by 0
  35. 2026The Attacker in the Mirror: Breaking Self-Consistency in Safety via Anchored Bipolicy Self-Playcited by 0
  36. 2026Same Model, Different Weakness: How Language and Modality Reshape the Jailbreak Attack Surface in Frontier MLLMscited by 0
  37. 2026Output Vector Editing for Memorization Mitigation in Large Language Modelscited by 0