mlsanityData to 14 Oct 2026

Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations

arXiv 2610.07115 ↗First posted 5 Oct 2026cs.LG cs.AIcited by 0

On the map: Mechanistic interpretability → Probing deception, bias and beliefs

Abstract (arXiv)

The order in which candidate responses are presented can change an LLM judge's verdict. Detecting such a position flip ordinarily requires judging each pair in both orders, which doubles the number of judgments. We investigate whether residual stream activations recorded immediately before the initial verdict can predict a flip. We use nested grouped cross-validation to evaluate regularized linear probes on 534 JudgeBench pairs for three Qwen3 judges and Llama-3.1-8B. The linear probes achieve AUROCs of .621-.850 and outperform a combined baseline that uses verbalized confidence, verdict-label logits, response lengths, and the judge's initial choice by .062-.113 AUROC. Linear probes trained on JudgeBench and then frozen achieve AUROCs of .685-.853 on 1,802 MT-Bench comparisons without MT-Bench fitting or recalibration. These results show that pre-verdict activations support prediction of susceptibility to candidate order and outperform the non-activation predictors evaluated here.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Probing deception, bias and beliefs, one of 13 directions in Mechanistic interpretability. Builds on 5 of its 12 references that are in the map. First posted on arXiv 5 Oct 2026.

Builds on

5 of 12 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimation23 Dec

Other references · 4

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2025Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representationscited by 15
  4. 2026Opening LLM Judges: Recovering Preference Signals Beyond the Final Verdictcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
  2. 2026VERDI: Single-Call Confidence Estimation for Verification-Based LLM Judges via Decomposed Inference
  3. 2026Jagged Judges: Epistemic Stability Under Perturbation, Pressure, and Persistence
  4. 2026LLM Judge Validation Under Sparse Overlap: From Inference to Design
  5. 2026Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models
  6. 2026Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias
  7. 2025Verdict: A Library for Scaling Judge-Time Compute
  8. 2026Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judges