mlsanityData to 13 Oct 2026

Hijacking Large Language Models via Adversarial In-Context Learning

arXiv 2311.09948 ↗First posted 16 Nov 2023cs.LG cs.CL cs.CRcited by 45

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

In-context learning (ICL) has emerged as a powerful paradigm leveraging LLMs for specific downstream tasks by utilizing labeled examples as demonstrations (demos) in the preconditioned prompts. Despite its promising performance, crafted adversarial attacks pose a notable threat to the robustness of LLMs. Existing attacks are either easy to detect, require a trigger in user input, or lack specificity towards ICL. To address these issues, this work introduces a novel transferable prompt injection attack against ICL, aiming to hijack LLMs to generate the target output or elicit harmful responses. In our threat model, the hacker acts as a model publisher who leverages a gradient-based prompt search method to learn and append imperceptible adversarial suffixes to the in-context demos via prompt injection. We also propose effective defense strategies using a few shots of clean demos, enhancing the robustness of LLMs during ICL. Extensive experimental results across various classification and jailbreak tasks demonstrate the effectiveness of the proposed attack and defense strategies. This work highlights the significant security vulnerabilities of LLMs during ICL and underscores the need for further in-depth studies.

Key facts. Cited by 45 papers in the map (data through 13 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 52 of its 88 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

52 of 88 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratch15 Jun
  2. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPT11 Apr
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2023GPT-4 Technical Reportcited by 18,278
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,884
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,064
  8. 2022Emergent Abilities of Large Language Modelscited by 2,499
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,319
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,424
  11. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,404
  12. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,376
37 more
  1. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,191
  2. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,126
  3. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 925
  4. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 817
  5. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 684
  6. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 647
  7. 2021Learning To Retrieve Prompts for In-Context Learningcited by 643
  8. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 520
  9. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 490
  10. 2023Detecting Language Model Attacks with Perplexitycited by 429
  11. 2023Larger language models do in-context learning differentlycited by 387
  12. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 381
  13. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 344
  14. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 335
  15. 2023Are aligned neural networks adversarially aligned?cited by 315
  16. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 277
  17. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 262
  18. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 260
  19. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 211
  20. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 192
  21. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  22. 2020Adversarial Training for Large Neural Language Modelscited by 110
  23. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Modelscited by 109
  24. 2023Backdoor Attacks for In-Context Learning with Language Modelscited by 96
  25. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 72
  26. 2023Adversarial Demonstration Attacks on Large Language Modelscited by 63
  27. 2023Black Box Adversarial Prompting for Foundation Modelscited by 62
  28. 2023In-context Example Selection with Influencescited by 60
  29. 2023In-Context Learning Learns Label Relationships but Is Not Conventional Learningcited by 50
  30. 2023When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Taskscited by 37
  31. 2023How Trustworthy are Open-Source LLMs? An Assessment under Malicious Demonstrations Shows their Vulnerabilitiescited by 33
  32. 2023Are Large Language Models Really Robust to Word-Level Perturbations?cited by 32
  33. 2023From Shortcuts to Triggers: Backdoor Defense with Denoised PoEcited by 27
  34. 2023Defending against Insertion-based Textual Backdoor Attacks via Attributioncited by 25
  35. 2020Improving Adversarial Robustness via Probabilistically Compact Loss with Logit Constraintscited by 10
  36. 2020Toward Tag-free Aspect Based Sentiment Analysis: A Multiple Attention Network Approachcited by 3
  37. 2022Learning Compact Features via In-Training Representation Alignmentcited by 2

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 2; 2024: 23; 2025: 10; 2026: 10.

45 papers.

Sort:

By area:

  1. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 664
  2. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 173
  3. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Modelscited by 106
  4. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 81
  5. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  6. 2024Instruction Backdoor Attacks Against Customized LLMscited by 61
  7. 2024Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learningcited by 53
  8. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  9. 2025Security Concerns for Large Language Models: A Surveycited by 29
  10. 2024Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attackscited by 26
  11. 2024Ranking Manipulation for Conversational Search Enginescited by 25
  12. 2024Data Poisoning for In-context Learningcited by 21
33 more
  1. 2024Efficient Universal Goal Hijacking with Semantics-guided Prompt Organizationcited by 16
  2. 2024Learning to Poison Large Language Models for Downstream Manipulationcited by 15
  3. 2024Cognitive Overload Attack:Prompt Injection for Long Contextcited by 13
  4. 2025X-Guard: Multilingual Guard Agent for Content Moderationcited by 13
  5. 2024Advancing Adversarial Suffix Transfer Learning on Aligned Large Language Modelscited by 10
  6. 2025Attention is All You Need to Defend Against Indirect Prompt Injection Attacks in LLMscited by 10
  7. 2024Diversity Helps Jailbreak Large Language Modelscited by 9
  8. 2024Merging Improves Self-Critique Against Jailbreak Attackscited by 8
  9. 2024Understanding In-Context Learning of Linear Models in Transformers Through an Adversarial Lenscited by 7
  10. 2025Unraveling LLM Jailbreaks Through Safety Knowledge Neuronscited by 7
  11. 2026When Personalization Legitimizes Risks: Uncovering Safety Vulnerabilities in Personalized Dialogue Agentscited by 7
  12. 2023Hijacking Context in Large Multi-modal Modelscited by 6
  13. 2025Not All Tokens Are Meant to Be Forgottencited by 6
  14. 2026Attention Smoothing Is All You Need For Unlearningcited by 6
  15. 2024Towards Action Hijacking of Large Language Model-based Agentcited by 4
  16. 2025On the Robustness of Transformers against Context Hijacking for Linear Classificationcited by 4
  17. 2024Soft Begging: Modular and Efficient Shielding of LLMs against Prompt Injection and Jailbreaking based on Prompt Tuningcited by 3
  18. 2024MILE: A Mutation Testing Framework of In-Context Learning Systemscited by 2
  19. 2025Automatic Calibration for Membership Inference Attack on Large Language Modelscited by 2
  20. 2025Securing Educational LLMs: A Generalised Taxonomy of Attacks on LLMs and DREAD Risk Assessmentcited by 2
  21. 2025How Can Mamba Learn In Context with Outliers and Generalize Provably?cited by 2
  22. 2026When AI reviews science: Can we trust the referee?cited by 2
  23. 2024MCGMark: An Encodable and Robust Online Watermark for Tracing LLM-Generated Malicious Codecited by 1
  24. 2024Defending against Jailbreak through Early Exit Generation of Large Language Modelscited by 1
  25. 2024Generative LLM Powered Conversational AI Application for Personalized Risk Assessment: A Case Study in COVID-19cited by 1
  26. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
  27. 2026Are Dilemmas and Conflicts in LLM Alignment Solvable? A View from Priority Graphcited by 1
  28. 2026Jailbreaking LLMs Without Gradients or Priors: Effective and Transferable Attackscited by 0
  29. 2026On the Hardness of Junking LLMscited by 0
  30. 2026Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMscited by 0
  31. 2026When Correct Demonstrations Hurt: Rethinking the Role of Exemplars in In-Context Learningcited by 0
  32. 2026Alignment of LRMs via Counter-Aligned Few-Shot Conversation Exposurecited by 0
  33. 2026Towards Mitigating Deceptive Safety Alignment in Large Reasoning Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluating and Safeguarding the Adversarial Robustness of Retrieval-Based In-Context Learning
  2. 2026Understanding and Improving Continuous Adversarial Training for LLMs via In-context Learning Theory
  3. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMs
  4. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations
  5. 2024Automatic and Universal Prompt Injection Attacks against Large Language Models
  6. 2024Human-Interpretable Adversarial Prompt Attack on Large Language Models with Situational Context
  7. 2025Breaking to Build: A Threat Model of Prompt-Based Attacks for Securing LLMs
  8. 2025OET: Optimization-based prompt injection Evaluation Toolkit