mlsanityData to 14 Oct 2026

Mitigating Rubric Interference in LLM Judges via On-Policy Self-Distillation

arXiv 2608.14684 ↗First posted 5 Aug 2026cs.LG cs.AIcited by 0

On the map: Knowledge distillation → On-policy and self-distillation for LLMs

Abstract (arXiv)

LLM judges increasingly evaluate responses against fine-grained rubric checklists. When a sample requires multiple rubrics, current methods typically assess each in a separate inference call. Evaluating all rubrics in a single pass is a natural alternative with greater efficiency, but we find that it introduces rubric interference: the verdict on one rubric shifts depending on which other rubrics are co-present. In a preliminary study, only one-third of samples receive fully consistent verdicts when evaluated under rubric sets of varying composition. We develop a measurement framework that probes interference through four controlled operations: rubric set expansion, subsetting, reordering, and noise injection. To mitigate interference without external supervision, we propose Self-Anchored Rubric Alignment (SARA). SARA uses a model's own single-rubric judgments as stable anchors and aligns multi-rubric reasoning with these anchors through on-policy self-distillation. We validate SARA on three datasets (HealthBench, FLASK, ResearchQA) and two model families (Qwen3, Llama-3.1). SARA consistently improves evaluation consistency while maintaining agreement with both base models and GPT-4.1 as a reference judge. Furthermore, the learned consistency transfers across datasets, confirming that SARA teaches a general capability rather than fitting dataset-specific patterns.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 22 of its 28 references that are in the map. First posted on arXiv 5 Aug 2026.

Builds on

22 of 28 references.

References · 22

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  4. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  5. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  6. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  7. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  8. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  9. 2018Born Again Neural Networkscited by 546
  10. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  11. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  12. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
10 more
  1. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  2. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  3. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  4. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
  5. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 205
  6. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 94
  7. 2025ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubricscited by 49
  8. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 43
  9. 2024CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judgescited by 34
  10. 2025Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubricscited by 29

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Rubrics from the Task, Verdicts from Evidence: Pointwise Rubric Rewards with Tool-Augmented Judges
  2. 2026Autorubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Tasks
  3. 2026Self-Preference Bias in Rubric-Based Evaluation of Large Language Models
  4. 2026Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?
  5. 2026Reliable to Expressive: A Curriculum for Rubric-Following Safety Judges
  6. 2026GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation
  7. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks
  8. 2025Online Rubrics Elicitation from Pairwise Comparisons