RELAY: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learning
On the map: LLM reasoning → Reinforcement learning for reasoning
also in 2 more
Abstract (arXiv)
Rubric-based reinforcement learning grades a response against many written criteria but collapses them into one reward, which discards the detail the rubric carries and gives no GRPO reward gradient when a group's responses tie. A frozen teacher that reads the rubric as privileged context provides token-level supervision, but we observe an early plateau in the gains over GRPO. RELAY uses advantages and rewards to select positive-advantage and low-scoring zero-advantage rollouts, and monitors the distillation loss to decide when to retire the teacher. Stage 1 combines selective distillation from a frozen teacher with GRPO; a plateau detector then removes the teacher, and Stage 2 continues with GRPO alone. Calibrated on one model-domain pair and reused unchanged in three others, the plateau detector fires between steps 122 and 171. RELAY scores highest in-domain among the compared methods in all four settings, ahead of GRPO on RaR-Medicine by 4.62 points with Qwen3-1.7B and 6.02 with Qwen3-4B. On Qwen3-1.7B/Medicine, removing the teacher from a shared checkpoint outperforms keeping, weakening or refreshing it. Distilling selected rollouts covering about half the response tokens also outperforms distilling all rollouts at a matched average coefficient.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 48 references that are in the map. First posted on arXiv 13 Oct 2026.
Builds on
36 of 48 references.
Closest ancestors · 8
Same direction, newest first
- 2026Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR3 Sep
- 2026CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization28 Jul
- 2026Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation17 Jun
- 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing2 Apr
- 2026PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalization27 Mar
- 2026Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMs20 Mar
- 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library6 Jun
- 2017Proximal Policy Optimization Algorithms20 Jul
Other references · 28
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
- 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
- 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
- 2026Reinforcement Learning via Self-Distillationcited by 332
- 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 299
- 2026On-Policy Context Distillation for Language Modelscited by 186
- 2026A Survey of On-Policy Distillation for Large Language Modelscited by 163
- 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 128
- 2025DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Researchcited by 114
- 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?cited by 103
- 2026RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generationcited by 52
16 more
- 2025ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubricscited by 49
- 2025Online Rubrics Elicitation from Pairwise Comparisonscited by 46
- 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixescited by 45
- 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillationcited by 45
- 2025LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validationcited by 34
- 2026STEPS: Selective On-Policy Self-Distillation for Reasoningcited by 16
- 2026Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVRcited by 15
- 2026Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifierscited by 15
- 2026Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidancecited by 11
- 2026Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Taskscited by 10
- 2026Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategycited by 6
- 2026CriPO: Enhancing Rubric-based RL via Self-Distillationcited by 5
- 2026Rubrics as Privileged Information for Open-Ended Generationcited by 4
- 2026PaperGym: Rubric-Centered Evolution for Research-Plan Generationcited by 3
- 2026RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learningcited by 3
- 2026OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillationcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization
- 2026Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL
- 2026MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learning
- 2026Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
- 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training
- 2026Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning
- 2026Reward Hacking in Rubric-Based Reinforcement Learning
- 2026CARE: Contrastive Anchor-based Rubric Evolution for Large Language Model Post-Training