mlsanityData to 7 Oct 2026

Adversarial Demonstration Attacks on Large Language Models

arXiv 2305.14950 ↗First posted 24 May 2023cs.CL cs.AI cs.CRcited by 58

Not on the map. Why?

Cited by 24 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

With the emergence of more powerful large language models (LLMs), such as ChatGPT and GPT-4, in-context learning (ICL) has gained significant prominence in leveraging these models for specific tasks by utilizing data-label pairs as precondition prompts. While incorporating demonstrations can greatly enhance the performance of LLMs across various tasks, it may introduce a new security concern: attackers can manipulate only the demonstrations without changing the input to perform an attack. In this paper, we investigate the security concern of ICL from an adversarial perspective, focusing on the impact of demonstrations. We propose a novel attack method named advICL, which aims to manipulate only the demonstration without changing the input to mislead the models. Our results demonstrate that as the number of demonstrations increases, the robustness of in-context learning would decrease. Additionally, we also identify the intrinsic property of the demonstrations is that they can be used (prepended) with different inputs. As a result, it introduces a more practical threat model in which an attacker can attack the test input example even without knowing and manipulating it. To achieve it, we propose the transferable version of advICL, named Transferable-advICL. Our experiment shows that the adversarial demonstration generated by Transferable-advICL can successfully attack the unseen test input examples. We hope that our study reveals the critical security risks associated with ICL and underscores the need for extensive research on the robustness of ICL, particularly given its increasing significance in the advancement of LLMs.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 26 of its 43 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

26 of 43 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  5. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  7. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  8. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  9. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  10. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  11. 2016Universal adversarial perturbationscited by 774
  12. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
14 more
  1. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  2. 2018Universal Sentence Encodercited by 485
  3. 2018Generating Natural Language Adversarial Examplescited by 422
  4. 2023Larger language models do in-context learning differentlycited by 386
  5. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  6. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
  7. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  8. 2018Stress Test Evaluation for Natural Language Inferencecited by 211
  9. 2019Certified Robustness to Adversarial Word Substitutionscited by 164
  10. 2019Combating Adversarial Misspellings with Robust Word Recognitioncited by 162
  11. 2016Crafting Adversarial Input Sequences for Recurrent Neural Networkscited by 102
  12. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 71
  13. 2022Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generatorcited by 68
  14. 2023In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPTcited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 21; 2025: 15; 2026: 8.

58 papers.

Sort:

By area:

  1. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  2. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  3. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  4. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  5. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  6. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  7. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  8. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  9. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 80
  10. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  11. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  12. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
38 more
  1. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Contentcited by 65
  2. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  3. 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflowscited by 57
  4. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  5. 2023Hijacking Large Language Models via Adversarial In-Context Learningcited by 41
  6. 2024Leveraging the Context through Multi-Round Interactions for Jailbreaking Attackscited by 32
  7. 2023MathAttack: Attacking Large Language Models Towards Math Solving Abilitycited by 31
  8. 2023DeceptPrompt: Exploiting LLM-driven Code Generation via Adversarial Natural Language Instructionscited by 31
  9. 2024Security of AI Agentscited by 24
  10. 2024Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-based Decision-Making Systemscited by 23
  11. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 21
  12. 2024Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Modelscited by 20
  13. 2024Preemptive Answer "Attacks" on Chain-of-Thought Reasoningcited by 16
  14. 2024Exploring the Robustness of In-Context Learning with Noisy Labelscited by 13
  15. 2025Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidencecited by 13
  16. 2025Attack and defense techniques in large language models: A survey and new perspectivescited by 13
  17. 2024Exploring Vulnerabilities and Protections in Large Language Models: A Surveycited by 10
  18. 2025Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformerscited by 10
  19. 2025A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experimentscited by 10
  20. 2024AdaPPA: Adaptive Position Pre-Fill Jailbreak Attack Approach Targeting LLMscited by 8
  21. 2023Hijacking Context in Large Multi-modal Modelscited by 6
  22. 2024Can LLMs be Fooled? Investigating Vulnerabilities in LLMscited by 5
  23. 2025LLM Agents Are Hypersensitive to Nudgescited by 5
  24. 2024Evaluating and Safeguarding the Adversarial Robustness of Retrieval-Based In-Context Learningcited by 4
  25. 2024Neutralizing Backdoors through Information Conflicts for Large Language Modelscited by 4
  26. 2025On the Robustness of Transformers against Context Hijacking for Linear Classificationcited by 4
  27. 2025Chain-of-Thought Driven Adversarial Scenario Extrapolation for Robust Language Modelscited by 4
  28. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  29. 2025A First Look at the Security Issues in the Model Context Protocol Ecosystemcited by 3
  30. 2023HANS, are you clever? Clever Hans Effect Analysis of Neural Systemscited by 2
  31. 2024MILE: A Mutation Testing Framework of In-Context Learning Systemscited by 2
  32. 2024The Best Defense is a Good Offense: Countering LLM-Powered Cyberattackscited by 2
  33. 2026Understanding the Dynamics of Demonstration Conflict in In-Context Learningcited by 2
  34. 2025Geometry-Guided Adversarial Prompt Detection via Curvature and Local Intrinsic Dimensioncited by 1
  35. 2025SoK: A Comprehensive Security Analysis of Jailbreak Resilience in GPT and DeepSeek Modelscited by 1
  36. 2026Visual Persuasion: What Influences Decisions of Vision-Language Models?cited by 1
  37. 2026Position: Behavioral Systems Require Behavioral Testscited by 1
  38. 2023Applying Standards to Advance Upstream & Downstream Ethics in Large Language Modelscited by 0

50 most cited of 58 All 58 (JSON)