mlsanityData to 7 Oct 2026

Optimization-based Prompt Injection Attack to LLM-as-a-Judge

arXiv 2403.17710 ↗First posted 26 Mar 2024cs.CR cs.AIcited by 123

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

also in 1 more
Abstract (arXiv)

LLM-as-a-Judge uses a large language model (LLM) to select the best response from a set of candidates for a given question. LLM-as-a-Judge has many applications such as LLM-powered search, reinforcement learning with AI feedback (RLAIF), and tool selection. In this work, we propose JudgeDeceiver, an optimization-based prompt injection attack to LLM-as-a-Judge. JudgeDeceiver injects a carefully crafted sequence into an attacker-controlled candidate response such that LLM-as-a-Judge selects the candidate response for an attacker-chosen question no matter what other candidate responses are. Specifically, we formulate finding such sequence as an optimization problem and propose a gradient based method to approximately solve it. Our extensive evaluation shows that JudgeDeceive is highly effective, and is much more effective than existing prompt injection attacks that manually craft the injected sequences and jailbreak attacks when extended to our problem. We also show the effectiveness of JudgeDeceiver in three case studies, i.e., LLM-powered search, RLAIF, and tool selection. Moreover, we consider defenses including known-answer detection, perplexity detection, and perplexity windowed detection. Our results show these defenses are insufficient, highlighting the urgent need for developing new defense strategies. Our implementation is available at this repository: https://github.com/ShiJiawenwen/JudgeDeceiver.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 81 references that are in the map. First posted on arXiv 26 Mar 2024.

Builds on

47 of 81 references.

References · 47

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023Mistral 7Bcited by 3,065
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  9. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  10. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  11. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
35 more
  1. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  2. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  3. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  4. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  5. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  6. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  7. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  8. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  9. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  10. 2023Detecting Language Model Attacks with Perplexitycited by 411
  11. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  12. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  13. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 325
  14. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  15. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  16. 2023Are aligned neural networks adversarially aligned?cited by 305
  17. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  18. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  19. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  20. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  21. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  22. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  23. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbotscited by 172
  24. 2023Generative Judge for Evaluating Alignmentcited by 163
  25. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agentscited by 135
  26. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 128
  27. 2021BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learningcited by 120
  28. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 91
  29. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  30. 2023BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPTcited by 70
  31. 2020Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERTcited by 69
  32. 2023Large Language Models in Education: Vision and Opportunitiescited by 54
  33. 2020Natural Backdoor Attack on Text Datacited by 18
  34. 2022Text Adversarial Purification as Defense against Adversarial Attackscited by 15
  35. 2023Backdoor Attacks to Pre-trained Unified Foundation Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 56; 2026: 54.

123 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  4. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 145
  5. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 119
  6. 2025PromptArmor: Simple yet Effective Prompt Injection Defensescited by 98
  7. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 93
  8. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 82
  9. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  10. 2025WebInject: Prompt Injection Attack to Web Agentscited by 48
  11. 2025VerIF: Verification Engineering for Reinforcement Learning in Instruction Followingcited by 43
  12. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 42
38 more
  1. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injectioncited by 42
  2. 2025Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifierscited by 40
  3. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 37
  4. 2024InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Modelscited by 36
  5. 2024Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applicationscited by 36
  6. 2025A Critical Evaluation of Defenses against Prompt Injection Attackscited by 31
  7. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  8. 2024Safeguarding Vision-Language Models Against Patched Visual Prompt Injectorscited by 26
  9. 2025PromptLocate: Localizing Prompt Injection Attackscited by 23
  10. 2026Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judgescited by 23
  11. 2025Les Dissonances: Cross-Tool Harvesting and Polluting in Pool-of-Tools Empowered LLM Agentscited by 19
  12. 2024Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignmentcited by 18
  13. 2024HonestLLM: Toward an Honest and Helpful Large Language Modelcited by 17
  14. 2025Beyond the Protocol: Unveiling Attack Vectors in the Model Context Protocol (MCP) Ecosystemcited by 17
  15. 2025Automatic Red Teaming LLM-based Agents with Model Context Protocol Toolscited by 17
  16. 2025ObliInjection: Order-Oblivious Prompt Injection Attack to LLM Agents with Multi-source Datacited by 17
  17. 2026MalTool: Malicious Tool Attacks on LLM Agentscited by 17
  18. 2025Improving Your Model Ranking on Chatbot Arena by Vote Riggingcited by 16
  19. 2025AudioJailbreak: Jailbreak Attacks against End-to-End Large Audio-Language Modelscited by 16
  20. 2026Overcoming the Retrieval Barrier: Indirect Prompt Injection in the Wild for LLM Systemscited by 16
  21. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  22. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challengescited by 14
  23. 2025BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judgecited by 13
  24. 2025Reassessing Extractive QA Datasets at Scale: LLM-as-a-Judge and In-Depth Analysescited by 13
  25. 2025Attack and defense techniques in large language models: A survey and new perspectivescited by 13
  26. 2025A Survey of Attacks on Large Language Modelscited by 13
  27. 2025PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Featurescited by 13
  28. 2026MCP-ITP: An Automated Framework for Implicit Tool Poisoning in MCPcited by 13
  29. 2025UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Modelscited by 12
  30. 2025Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMscited by 12
  31. 2026BiasScope: Towards Automated Detection of Bias in LLM-as-a-Judge Evaluationcited by 12
  32. 2025ControlNET: A Firewall for RAG-based LLM Systemcited by 11
  33. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 11
  34. 2025Adaptive Distraction: Probing LLM Contextual Robustness with Automated Tree Searchcited by 10
  35. 2025From Assistants to Adversaries: Exploring the Security Risks of Mobile LLM Agentscited by 10
  36. 2025Secure and Efficient Access Control for Computer-Use Agents via Context Spacecited by 10
  37. 2025IP Leakage Attacks Targeting LLM-Based Multi-Agent Systemscited by 8
  38. 2025Fingerprinting LLMs via Prompt Injectioncited by 8

50 most cited of 123 All 123 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Are My Optimized Prompts Compromised? Exploring Vulnerabilities of LLM-based Optimizers
  2. 2026Prompt Attack Detection with LLM-as-a-Judge and Mixture-of-Models
  3. 2025Improve LLM-as-a-Judge Ability as a General Ability
  4. 2025Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency Through Chain-of-Thought Fine-Tuning and Alignment
  5. 2025Defend LLMs Through Self-Consciousness
  6. 2026D-Judge: Disrupting Multi-Turn Jailbreaks using Semantics-Preserving Output Rewriting
  7. 2025Efficient LLM Safety Evaluation through Multi-Agent Debate
  8. 2024Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs