mlsanityData to 7 Oct 2026

PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts

arXiv 2306.04528 ↗First posted 7 Jun 2023cs.CL cs.CR cs.LGcited by 218

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

The increasing reliance on Large Language Models (LLMs) across academia and industry necessitates a comprehensive understanding of their robustness to prompts. In response to this vital need, we introduce PromptRobust, a robustness benchmark designed to measure LLMs' resilience to adversarial prompts. This study uses a plethora of adversarial textual attacks targeting prompts across multiple levels: character, word, sentence, and semantic. The adversarial prompts, crafted to mimic plausible user errors like typos or synonyms, aim to evaluate how slight deviations can affect LLM outcomes while maintaining semantic integrity. These prompts are then employed in diverse tasks including sentiment analysis, natural language inference, reading comprehension, machine translation, and math problem-solving. Our study generates 4,788 adversarial prompts, meticulously evaluated over 8 tasks and 13 datasets. Our findings demonstrate that contemporary LLMs are not robust to adversarial prompts. Furthermore, we present a comprehensive analysis to understand the mystery behind prompt robustness and its transferability. We then offer insightful robustness analysis and pragmatic recommendations for prompt composition, beneficial to both researchers and everyday users.

Key facts. Cited by 218 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 42 of its 83 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

42 of 83 references.

References · 42

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  10. 2016Adversarial examples in the physical worldcited by 1,762
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
30 more
  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  2. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  3. 2018Neural Network Acceptability Judgmentscited by 952
  4. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  5. 2021Robust fine-tuning of zero-shot modelscited by 785
  6. 2016Universal adversarial perturbationscited by 774
  7. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  8. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  9. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  10. 2017Adversarial Patchcited by 514
  11. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  12. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  13. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  14. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
  15. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  16. 2022Data Contamination: From Memorization to Exploitationcited by 234
  17. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  18. 2019Improving Adversarial Robustness via Promoting Ensemble Diversitycited by 213
  19. 2018Stress Test Evaluation for Natural Language Inferencecited by 211
  20. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  21. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
  22. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  23. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  24. 2017Data Augmentation for Low-Resource Neural Machine Translationcited by 139
  25. 2017Bilateral Multi-Perspective Matching for Natural Language Sentencescited by 120
  26. 2017Personalizing Session-based Recommendations with Hierarchical Recurrent Neural Networkscited by 84
  27. 2023Safety Assessment of Chinese Large Language Modelscited by 82
  28. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
  29. 2023Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physicscited by 49
  30. 2023On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codexcited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 30; 2024: 62; 2025: 61; 2026: 65.

218 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  3. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  4. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  5. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  6. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
  7. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  8. 2023How Robust is Google's Bard to Adversarial Image Attacks?cited by 177
  9. 2023Large Language Models Understand and Can be Enhanced by Emotional Stimulicited by 143
  10. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  11. 2023CMB: A Comprehensive Medical Benchmark in Chinesecited by 113
  12. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 102
38 more
  1. 2023Identifying and Mitigating the Security Risks of Generative AIcited by 90
  2. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 77
  3. 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competitioncited by 65
  4. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  5. 2025Authenticated Delegation and Authorized AI Agentscited by 63
  6. 2024Dynamic Evaluation of Large Language Models by Meta Probing Agentscited by 54
  7. 2024Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compressioncited by 54
  8. 2023Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injectioncited by 53
  9. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 51
  10. 2024Measuring Implicit Bias in Explicitly Unbiased Large Language Modelscited by 49
  11. 2024VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulationcited by 43
  12. 2024A Cross-Language Investigation into Jailbreak Attacks in Large Language Modelscited by 37
  13. 2024On the Worst Prompt Performance of Large Language Modelscited by 36
  14. 2023GameEval: Evaluating LLMs on Conversational Gamescited by 35
  15. 2025Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMscited by 34
  16. 2023How Trustworthy are Open-Source LLMs? An Assessment under Malicious Demonstrations Shows their Vulnerabilitiescited by 33
  17. 2023Foundation Metrics for Evaluating Effectiveness of Healthcare Conversations Powered by Generative AIcited by 32
  18. 2023MathAttack: Attacking Large Language Models Towards Math Solving Abilitycited by 31
  19. 2024On the Challenges and Opportunities in Generative AIcited by 31
  20. 2024Examining the robustness of LLM evaluation to the distributional assumptions of benchmarkscited by 31
  21. 2024Reasoning Robustness of LLMs to Adversarial Typographical Errorscited by 31
  22. 2024Prompt Perturbation Consistency Learning for Robust Language Modelscited by 26
  23. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputscited by 25
  24. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directionscited by 25
  25. 2023Beyond Static Datasets: A Deep Interaction Approach to LLM Evaluationcited by 24
  26. 2023Fool Your (Vision and) Language Model With Embarrassingly Simple Permutationscited by 24
  27. 2024UltraEval: A Lightweight Platform for Flexible and Comprehensive Evaluation for LLMscited by 24
  28. 2023Intuitive or Dependent? Investigating LLMs' Behavior Style to Conflicting Promptscited by 23
  29. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  30. 2025Beyond Prompt Engineering: Robust Behavior Control in LLMs via Steering Target Atomscited by 23
  31. 2024Benchmarking Large Multimodal Models against Common Corruptionscited by 22
  32. 2024From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreakingcited by 22
  33. 2024Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let's Take TravelPlanner as an Examplecited by 22
  34. 2023LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarkingcited by 20
  35. 2023METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualitiescited by 20
  36. 2024Resilience of Large Language Models for Noisy Instructionscited by 20
  37. 2024Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Modelscited by 20
  38. 2024Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Taskscited by 20

50 most cited of 218 All 218 (JSON)