mlsanityData to 7 Oct 2026

One Token to Fool LLM-as-a-Judge

arXiv 2507.08794 ↗First posted 11 Jul 2025cs.LG cs.CLcited by 56

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Large language models (LLMs) are increasingly trusted as automated judges, assisting evaluation and providing reward signals for training other models, particularly in reference-based settings like Reinforcement Learning with Verifiable Rewards (RLVR). However, we uncover a critical vulnerability even in this reference-based paradigm: generative reward models are systematically susceptible to reward hacking. We find that superficial inputs, which we term ''master keys'' such as non-word symbols (e.g., '':'' or ''.'') or generic reasoning openers (e.g., ''Thought process:'' or ''Let's solve this problem step by step.''), can consistently elicit false positive rewards without any substantive reasoning. Our systematic evaluation demonstrates this is a widespread failure affecting a diverse range of models, including leading proprietary systems such as GPT-o1 and Claude-4. These results challenge the assumed robustness of LLM judges and pose a significant threat to their reliability. To address this, we propose a simple yet effective data augmentation strategy using truncated model outputs as adversarial negative examples. The resulting Master Reward Models (Master-RMs) demonstrate state-of-the-art robustness against these ''master key'' attacks while maintaining high performance in standard evaluation settings. We supplement these findings with a comprehensive analysis of the vulnerability across model scales, prompt variations, and common inference-time strategies, offering insights to guide future research on robust LLM evaluation. We release our robust, general-domain reward models and the synthetic training data at https://huggingface.co/sarosavo/Master-RM and https://huggingface.co/datasets/sarosavo/Master-RM.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 67 references that are in the map. First posted on arXiv 11 Jul 2025.

Builds on

44 of 67 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains31 Mar
  2. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug

Other references · 42

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  8. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  9. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  10. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  11. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  12. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
30 more
  1. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
  2. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  3. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  4. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 260
  5. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  6. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  7. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
  8. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 159
  9. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  10. 2024MAmmoTH2: Scaling Instructions from the Webcited by 143
  11. 2024Stream of Search (SoS): Learning to Search in Languagecited by 141
  12. 2025General-Reasoner: Advancing LLM Reasoning Across All Domainscited by 132
  13. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 123
  14. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 119
  15. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  16. 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuningcited by 106
  17. 2024Rule Based Rewards for Language Model Safetycited by 106
  18. 2024InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationalescited by 100
  19. 2025NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questionscited by 81
  20. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 42
  21. 2025On the Trustworthiness of Generative Foundation Models: Guideline, Assessment, and Perspectivecited by 39
  22. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  23. 2025Assessing Judging Bias in Large Reasoning Models: An Empirical Studycited by 30
  24. 2024Defending Jailbreak Prompts via In-Context Adversarial Gamecited by 27
  25. 2025SoRFT: Issue Resolving with Subtask-oriented Reinforced Fine-Tuningcited by 27
  26. 2025Learning to Reason via Mixture-of-Thought for Logical Reasoningcited by 20
  27. 2025VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Modelscited by 19
  28. 2024OpenRFT: Adapting Reasoning Foundation Model for Domain-specific Tasks with Reinforcement Fine-Tuningcited by 13
  29. 2024How Do Humans Write Code? Large Models Do It the Same Way Toocited by 11
  30. 2021Self-Teaching Machines to Read and Comprehend with Large-Scale Multi-Subject Question-Answering Datacited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 20; 2026: 36.

56 papers.

Sort:

By area:

  1. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  2. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 44
  3. 2025VisPlay: Self-Evolving Vision-Language Models from Imagescited by 42
  4. 2025Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifierscited by 40
  5. 2025Guided Self-Evolving LLMs with Minimal Human Supervisioncited by 33
  6. 2025CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Modelscited by 28
  7. 2026SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Modelscited by 20
  8. 2025Beyond Consensus: Mitigating the Agreeableness Bias in LLM Judge Evaluationscited by 16
  9. 2026MM-Zero: Self-Evolving Multi-Model Vision Language Models From Zero Datacited by 16
  10. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 11
  11. 2025Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradientscited by 11
  12. 2026Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoningcited by 10
38 more
  1. 2026CARE: Confounder-Aware Aggregation for Reliable LLM Evaluationcited by 10
  2. 2025PersonaEval: Are LLM Evaluators Human Enough to Judge Role-Play?cited by 9
  3. 2026Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learningcited by 9
  4. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewardscited by 8
  5. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  6. 2025SCI-Verifier: Scientific Verifier with Thinkingcited by 6
  7. 2025Who's Your Judge? On the Detectability of LLM-Generated Judgmentscited by 6
  8. 2025FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoningcited by 6
  9. 2026Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluationcited by 6
  10. 2026Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVRcited by 6
  11. 2026More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judgescited by 6
  12. 2025Enhancing Radiology Report Generation and Visual Grounding using Reinforcement Learningcited by 4
  13. 2026GenArena: How Can We Achieve Human-Aligned Evaluation for Visual Generation Tasks?cited by 4
  14. 2026DeepInnovator: Triggering the Innovative Capabilities of LLMscited by 4
  15. 2026Automated structural testing of LLM-based agents: methods, framework, and case studiescited by 3
  16. 2026Beyond Semantic Manipulation: Token-Space Attacks on Reward Modelscited by 3
  17. 2025Becoming Experienced Judges: Selective Test-Time Learning for Evaluatorscited by 2
  18. 2025AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokenscited by 2
  19. 2026Distribution-Aware Reward Estimation for Test-Time Reinforcement Learningcited by 2
  20. 2026Small Reward Models via Backward Inferencecited by 2
  21. 2026Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Datacited by 2
  22. 2026When AI reviews science: Can we trust the referee?cited by 2
  23. 2025Better Language Model-Based Judging Reward Modeling through Scaling Comprehension Boundariescited by 1
  24. 2025Every Question Has Its Own Value: Reinforcement Learning with Explicit Human Valuescited by 1
  25. 2026A Primer in Post-Training Reasoning Data: What We Know About How It Workscited by 1
  26. 2026Agreement Overstates Evidence: Error Dependence in LLM Judge Consensuscited by 1
  27. 2026JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Modelscited by 1
  28. 2025Gaming the Answer Matcher: Examining the Impact of Text Manipulation on Automated Judgmentcited by 0
  29. 2026Making Bias Non-Predictive: Training Robust LLM Reasoning via Reinforcement Learningcited by 0
  30. 2026Blind to the Human Touch: Overlap Bias in LLM-Based Summary Evaluationcited by 0
  31. 2026Alignment Risks from Capability-Seeking RL Trainingcited by 0
  32. 2026Suffix-Constrained Greedy Search Algorithms for Causal Language Modelscited by 0
  33. 2026MoRI: Learning Motivation-Grounded Reasoning for Scientific Ideation in Large Language Modelscited by 0
  34. 2026LLM-as-Judge for Semantic Judging of Powerline Segmentation in UAV Inspectioncited by 0
  35. 2026JURY-RL: Votes Propose, Proofs Dispose for Label-Free RLVRcited by 0
  36. 2026Likelihood scoring for continuations of mathematical text: a self-supervised benchmark with tests for shortcut vulnerabilitiescited by 0
  37. 2026Provably Secure Agent Guardrailcited by 0
  38. 2026Codifying the Judge: Scalable Evaluation via Program Distillationcited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reverse Engineering Human Preferences with Reinforcement Learning
  2. 2025Confidence as a Reward: Transforming LLMs into Reward Models
  3. 2025Explicit Reasoning Makes Better Judges: A Systematic Study on Accuracy, Efficiency, and Robustness
  4. 2026On the effectiveness of reward functions in reinforcement learning for confidence calibration of large language models
  5. 2025Inference-Time Reward Hacking in Large Language Models
  6. 2026Likelihood-Based Reward Designs for General LLM Reasoning
  7. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
  8. 2024LLM-as-a-Judge & Reward Model: What They Can and Cannot Do