mlsanityData to 7 Oct 2026

Training LLM Judges from Language Feedback via Position-Selective Self-Distillation

arXiv 2609.38792 ↗First posted 30 Sep 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We study training LLM judges from natural language feedback, especially for subjective tasks where the verdict depends strongly on which evaluation criteria the judge invokes and how it weighs them. The dominant approach, outcome-supervised RL (e.g., GRPO), credits every token in the rollout with a single scalar determined only by the accuracy of the final verdict, providing no separate credit at the criterion-choice tokens and ignoring the rich language feedback (e.g., preference rationales) that naturally accompanies preference labels. Self-Distillation (SD) is one natural way to use this language feedback: the same model, conditioned on this feedback, acts as a teacher providing dense, position-level supervision. However, not all positions carry equally useful signal. Using the per-position entropy shift between teacher and student, we identify two regimes: context sharpening, where the teacher concentrates probability on a particular feedback-aligned criterion expression, and context spreading, where the teacher distributes probability across multiple feedback-aligned alternatives. We interpret these patterns as follows: sharpening encourages memorization of a particular criterion expression, whereas spreading promotes semantic understanding by preserving these alternatives. Motivated by this asymmetry, we introduce position masking based on the entropy shift that retains the lower tail of the entropy-shift distribution. Experiments show that masking higher-entropy-shift positions improves out-of-distribution generalization over naive SD. The resulting self-distilled judges outperform judges trained with outcome-supervised RL by 2-9 percentage points on the evaluated subjective subcategories, while remaining competitive on objective ones.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 45 of its 51 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

45 of 51 references.

References · 45

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  8. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  9. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  10. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  11. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  12. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
33 more
  1. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  2. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  3. 2026Reinforcement Learning via Self-Distillationcited by 285
  4. 2026Self-Distillation Enables Continual Learningcited by 212
  5. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  6. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  7. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  8. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  9. 2024Token-level Direct Preference Optimizationcited by 135
  10. 2023Training Language Models with Language Feedback at Scalecited by 125
  11. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  12. 2026TIP: Token Importance in On-Policy Distillationcited by 90
  13. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?cited by 87
  14. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedbackcited by 86
  15. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agentscited by 75
  16. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
  17. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
  18. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 55
  19. 2025Reward Reasoning Modelcited by 47
  20. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  21. 2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weightscited by 42
  22. 2025Feedback Descent: Open-Ended Text Optimization via Pairwise Comparisoncited by 42
  23. 2025Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuningcited by 39
  24. 2026Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Levelcited by 29
  25. 2024Learning to Refine with Fine-Grained Natural Language Feedbackcited by 27
  26. 2026When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoningcited by 24
  27. 2025Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuningcited by 20
  28. 2025Text2Grad: Reinforcement Learning from Natural Language Feedbackcited by 18
  29. 2025Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Modelscited by 15
  30. 2025Token-Importance Guided Direct Preference Optimizationcited by 14
  31. 2026Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Modelscited by 9
  32. 2026Training-Trajectory-Aware Token Selectioncited by 3
  33. 2026GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Role of Feedback Alignment in Self-Distillation
  2. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
  3. 2024Self-Judge: Selective Instruction Following with Alignment Self-Evaluation
  4. 2026RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges
  5. 2024Aligning Large Language Models by On-Policy Self-Judgment
  6. 2026Rethinking Selective Knowledge Distillation
  7. 2026RTLC -- Research, Teach-to-Learn, Critique: A three-stage prompting paradigm inspired by the Feynman Learning Technique that lifts LLM-as-judge accuracy on JudgeBench with no fine-tuning
  8. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators