# Can You Trick the Grader? Adversarial Persuasion of LLM Judges

arXiv: https://arxiv.org/abs/2508.07805 · HTML: https://mlsanity.com/paper/2508.07805/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 39 references that are in the map.
- First posted on arXiv 11 Aug 2025.

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs](https://mlsanity.com/paper/2401.06373/) (2024)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [Can LLM be a Personalized Judge?](https://mlsanity.com/paper/2406.11657/) (2024)
- [Turning large language models into cognitive models](https://mlsanity.com/paper/2306.03917/) (2023)
- [MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation](https://mlsanity.com/paper/2312.17080/) (2023)
- [EXAONE 3.5: Series of Large Language Models for Real-world Use Cases](https://mlsanity.com/paper/2412.04862/) (2024)
- [Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators](https://mlsanity.com/paper/2504.15253/) (2025)
- [Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluation](https://mlsanity.com/paper/2410.20774/) (2024)
- [Benchmarking Adversarial Robustness to Bias Elicitation in Large Language Models: Scalable Automated Assessment with LLM-as-a-Judge](https://mlsanity.com/paper/2504.07887/) (2025)
- [From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Tasks](https://mlsanity.com/paper/2409.04168/) (2024)
- [Scaling Trends in Language Model Robustness](https://mlsanity.com/paper/2407.18213/) (2024)
- [LLMs can be easily Confused by Instructional Distractions](https://mlsanity.com/paper/2502.04362/) (2025)
- [Evidence of a log scaling law for political persuasion with large language models](https://mlsanity.com/paper/2406.14508/) (2024)

## Built on by (most cited first, 9 of 9)

- [When Wording Steers the Evaluation: Framing Bias in LLM judges](https://mlsanity.com/paper/2601.13537/) (2026)
- [Judging Against the Reference: Uncovering Knowledge-Driven Failures in LLM-Judges on QA Evaluation](https://mlsanity.com/paper/2601.07506/) (2026)
- [Aletheia: What Makes RLVR For Code Verifiers Tick?](https://mlsanity.com/paper/2601.12186/) (2026)
- [Sell More, Play Less: Benchmarking LLM Realistic Selling Skill](https://mlsanity.com/paper/2604.07054/) (2026)
- [Silence Is Endorsement: Verification-Status Laundering in LLM Agent Pipelines](https://mlsanity.com/paper/2609.20211/) (2026)
- [FinReflectKG -- EvalBench: Benchmarking Financial KG with Multi-Dimensional Evaluation](https://mlsanity.com/paper/2510.05710/) (2025)
- [Beyond the Resum'e: A Rubric-Aware Automatic Interview System for Information Elicitation](https://mlsanity.com/paper/2603.01775/) (2026)
- [Black-Box Auditing of Epistemic Reliability in Multi-Agent Debate Distillation](https://mlsanity.com/paper/2609.32361/) (2026)
- [Breaking the Illusion of Review Reliability under Static Evaluation: SCOPE Fuzzing for LLM-based Scientific Reviewers](https://mlsanity.com/paper/2609.37097/) (2026)

## Abstract (arXiv)

As large language models take on growing roles as automated evaluators in practical settings, a critical question arises: Can individuals persuade an LLM judge to assign unfairly high scores? This study is the first to reveal that strategically embedded persuasive language can bias LLM judges when scoring mathematical reasoning tasks, where correctness should be independent of stylistic variation. Grounded in Aristotle's rhetorical principles, we formalize seven persuasion techniques (Majority, Consistency, Flattery, Reciprocity, Pity, Authority, Identity) and embed them into otherwise identical responses. Across six math benchmarks, we find that persuasive language leads LLM judges to assign inflated scores to incorrect solutions, by up to 8% on average, with Consistency causing the most severe distortion. Notably, increasing model size does not substantially mitigate this vulnerability. Further analysis demonstrates that combining multiple persuasion techniques amplifies the bias, and pairwise evaluation is likewise susceptible. Moreover, the persuasive effect persists under counter prompting strategies, highlighting a critical vulnerability in LLM-as-a-Judge pipelines and underscoring the need for robust defenses against persuasion-based attacks.