mlsanityData to 7 Oct 2026

Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment

arXiv 2402.14016 ↗First posted 21 Feb 2024cs.CLcited by 117

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are powerful zero-shot assessors used in real-world situations such as assessing written exams and benchmarking systems. Despite these critical applications, no existing work has analyzed the vulnerability of judge-LLMs to adversarial manipulation. This work presents the first study on the adversarial robustness of assessment LLMs, where we demonstrate that short universal adversarial phrases can be concatenated to deceive judge LLMs to predict inflated scores. Since adversaries may not know or have access to the judge-LLMs, we propose a simple surrogate attack where a surrogate model is first attacked, and the learned attack phrase then transferred to unknown judge-LLMs. We propose a practical algorithm to determine the short universal attack phrases and demonstrate that when transferred to unseen models, scores can be drastically inflated such that irrespective of the assessed text, maximum scores are predicted. It is found that judge-LLMs are significantly more susceptible to these adversarial attacks when used for absolute scoring, as opposed to comparative assessment. Our findings raise concerns on the reliability of LLM-as-a-judge methods, and emphasize the importance of addressing vulnerabilities in LLM assessment methods before deployment in high-stakes real-world scenarios.

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). Builds on 39 of its 57 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

39 of 57 references.

References · 39

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2013Intriguing properties of neural networkscited by 5,114
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  8. 2020Extracting Training Data from Large Language Modelscited by 2,144
  9. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  11. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  12. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
27 more
  1. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  2. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  3. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  4. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  5. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 530
  6. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  7. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  8. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  9. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  10. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  11. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  12. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  13. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
  14. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  15. 2023Are aligned neural networks adversarially aligned?cited by 305
  16. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 302
  17. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  18. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  19. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 156
  20. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  21. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  22. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 91
  23. 2023Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversationscited by 88
  24. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  25. 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
  26. 2023MQAG: Multiple-choice Question Answering and Generation for Assessing Information Consistency in Summarizationcited by 30
  27. 2018Why Do Adversarial Attacks Transfer? Explaining Transferability of Evasion and Poisoning Attackscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 44; 2026: 47.

117 papers.

Sort:

By area:

  1. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  3. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  4. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  5. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  6. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 80
  7. 2025One Token to Fool LLM-as-a-Judgecited by 56
  8. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  9. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 42
  10. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 37
  11. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 36
  12. 2025Investigating Non-Transitivity in LLM-as-a-Judgecited by 33
38 more
  1. 2024CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judgescited by 32
  2. 2024Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluationcited by 29
  3. 2025How Reliable is Multilingual LLM-as-a-Judge?cited by 29
  4. 2025Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifactscited by 25
  5. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputscited by 25
  6. 2026UniCorn: Towards Self-Improving Unified Multimodal Models through Self-Generated Supervisioncited by 23
  7. 2024From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Taskscited by 22
  8. 2025TripTailor: A Real-World Benchmark for Personalized Travel Planningcited by 22
  9. 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluationscited by 20
  10. 2024Understanding Figurative Meaning through Explainable Visual Entailmentcited by 19
  11. 2024On Large Language Models in National Security Applicationscited by 18
  12. 2024CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMscited by 18
  13. 2024Empirical Guidelines for Deploying LLMs onto Resource-constrained Edge Devicescited by 17
  14. 2025Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMscited by 17
  15. 2024Grade Like a Human: Rethinking Automated Assessment with Large Language Modelscited by 16
  16. 2024PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining Taskscited by 16
  17. 2025Improving Your Model Ranking on Chatbot Arena by Vote Riggingcited by 16
  18. 2025Position: Adversarial ML for LLMs Is Not Making Any Progresscited by 14
  19. 2025Breaking the Reviewer: Assessing the Vulnerability of Large Language Models in Automated Peer Review Under Textual Adversarial Attackscited by 14
  20. 2024Cognitive Overload Attack:Prompt Injection for Long Contextcited by 13
  21. 2025BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judgecited by 13
  22. 2025Reassessing Extractive QA Datasets at Scale: LLM-as-a-Judge and In-Depth Analysescited by 13
  23. 2026Agent-Sentry: Bounding LLM Agents via Execution Provenancecited by 13
  24. 2026Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJackcited by 13
  25. 2025UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Modelscited by 12
  26. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  27. 2024Towards Understanding the Robustness of LLM-based Evaluations under Perturbationscited by 10
  28. 2025Fooling the LVLM Judges: Visual Biases in LVLM-Based Evaluationcited by 10
  29. 2026Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judgescited by 8
  30. 2024MCRanker: Generating Diverse Criteria On-the-Fly to Improve Point-wise LLM Rankerscited by 7
  31. 2025UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluationcited by 7
  32. 2025TAI3: Testing Agent Integrity in Interpreting User Intentcited by 7
  33. 2025Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankingscited by 7
  34. 2025SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoningcited by 7
  35. 2025Can LLMs Narrate Tabular Data? An Evaluation Framework for Natural Language Representations of Text-to-SQL System Outputscited by 7
  36. 2026Omni-R1: Towards the Unified Generative Paradigm for Multimodal Reasoningcited by 7
  37. 2024Data Analysis in the Era of Generative AIcited by 6
  38. 2024Attack-in-the-Chain: Bootstrapping Large Language Models for Attacks Against Black-box Neural Ranking Modelscited by 6

50 most cited of 117 All 117 (JSON)