mlsanityData to 7 Oct 2026

AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokens

arXiv 2512.17375 ↗First posted 19 Dec 2025cs.LG cs.CL cs.CRcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

LLM-as-a-Judge systems supply the reward signal in modern RLHF and RLVR pipelines, but their binary verdict reduces to a single linear readout Fgap on one hidden state. We show this readout is shallow enough that short, low-perplexity tokens flip the verdict from "No" to "Yes". These tokens are sampled from the judge's own next-token distribution at the response position, with no manual seed set and no gradient-based optimization. Our procedure, AdvJudge-Zero, reaches >90% ensemble false-positive rate on 22 of 24 (model, dataset) cells across six Qwen, Llama, and Gemma judges, versus 54-72% for the prior curated 10-token benchmark, and the discovered surface transfers cross-format to a 70B scalar reward model. The same discovered pool enables a defense: a LoRA fine-tune stratified by a 9-class mechanism taxonomy hardens cross-family generalization where naive sampling on the same pool fails, with mechanism breadth rather than pool size carrying the gain. Under GRPO training, the hardened judge eliminates the reward-collapse failures (false-positive spikes and length collapse) we observe in the unhardened baseline on both MATH and GSM8K at ten seeds per condition. The discovered pool, the mechanism taxonomy, and per-prompt flip records will be released under responsible disclosure.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 36 references that are in the map. First posted on arXiv 19 Dec 2025.

Builds on

27 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Adversarial Training of Reward Models8 Apr

Other references · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2024Qwen2 Technical Reportcited by 2,076
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  12. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
14 more
  1. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  2. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  3. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domainscited by 150
  4. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  5. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedbackcited by 86
  6. 2024Building Guardrails for Large Language Modelscited by 61
  7. 2025One Token to Fool LLM-as-a-Judgecited by 56
  8. 2025The Path Not Taken: RLVR Provably Learns Off the Principalscited by 52
  9. 2025Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Modelscited by 48
  10. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 47
  11. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 37
  12. 2025Language Models are Injective and Hence Invertiblecited by 31
  13. 2025Shrinking the Generation-Verification Gap with Weak Verifierscited by 21
  14. 2025Debiasing Reward Models by Representation Learning with Guaranteescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026PReMISE: Policy Rubrics as Measurement Specifications for LLM Judgescited by 2
  2. 2026Adversarial Cues in Decision Models Used as Judges: The Role of Request Presentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judge
  2. 2026Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judges
  3. 2026Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training
  4. 2026Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoning
  5. 2026LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning
  6. 2025Reverse Engineering Human Preferences with Reinforcement Learning
  7. 2026Right in the Right Way: LM Training with Verifiable Rewards and Human Demonstrations
  8. 2026MARS: Margin-Adversarial Risk-controlled Stopping for Parallel LLM Test-time Scaling