mlsanityData to 7 Oct 2026

Reasoning Model Is Superior LLM-Judge, Yet Suffers from Biases

arXiv 2601.03630 ↗First posted 7 Jan 2026cs.CLcited by 1

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

This paper presents the first systematic comparison investigating whether Large Reasoning Models (LRMs) are superior judges to non-reasoning LLMs. Our empirical analysis yields four key findings: 1) LRMs outperform non-reasoning LLMs in terms of judgment accuracy, particularly on reasoning-intensive tasks; 2) LRMs demonstrate superior evaluation instruction-following capabilities; 3) LRMs exhibit enhanced robustness against adversarial attacks targeting judgment tasks; 4) However, LRMs still exhibit strong evaluation biases. To mitigate this bias vulnerability, we propose PlanJudge, a lightweight evaluation strategy that prompts the model to generate an explicit evaluation plan before executing the judgment. Despite its simplicity, our experiments demonstrate that PlanJudge significantly mitigates biases in LLM-as-a-Judge while preserving overall judgment accuracy.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 13 of its 22 references that are in the map. First posted on arXiv 7 Jan 2026.

Builds on

13 of 22 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Qwen3 Technical Report14 May
  2. 2025Assessing Judging Bias in Large Reasoning Models: An Empirical Study14 Apr
  3. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge30 Jan

Other references · 10

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 298
  3. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  4. 2025Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMscited by 115
  5. 2025Are Reasoning Models More Prone to Hallucination?cited by 62
  6. 2025When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMscited by 51
  7. 2025LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judgecited by 33
  8. 2025Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Modelscited by 28
  9. 2025Reasoning Model is Stubborn: Diagnosing Instruction Overriding in Reasoning Modelscited by 6
  10. 2025Policy-based Sentence Simplification: Replacing Parallel Corpora with LLM-as-a-Judgecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Evidence Lock Before Commitment: A Frozen Interface Degrades LLM-as-Judge Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training
  2. 2024From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Tasks
  3. 2025Explicit Reasoning Makes Better Judges: A Systematic Study on Accuracy, Efficiency, and Robustness
  4. 2025Verdict: A Library for Scaling Judge-Time Compute
  5. 2025ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning
  6. 2025JudgeLRM: Large Reasoning Models as a Judge
  7. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judges
  8. 2026Measuring Reasoning Quality in LLMs: A Multi-Dimensional Behavioral Framework