mlsanityData to 7 Oct 2026

Reinforcement Learning-based Semi-supervised Knowledge Distillation with LLM-as-a-Judge

arXiv 2604.02621 ↗First posted 3 Apr 2026cs.CL cs.LGcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning (RL) substantially improves the reasoning capabilities of language models, but most existing RL fine-tuning approaches rely entirely on ground-truth verifiable rewards and thus labeled datasets with verifiable answers. To overcome this, we propose a RL framework for reasoning distillation that leverages continuous, LLM-based rewards. Our method employs an efficient mechanism that computes a continuous CoT reward (CCR) directly from a single-token logit of a judge LLM, evaluating the student model's reasoning trajectory. This formulation provides an effective and scalable online training signal that can be applied to large volumes of unlabeled data. We demonstrate that, when paired with a strong judge, simply using CCR achieves performance comparable to that of ground-truth or pseudo-label verifiable rewards, and even surpasses them as the amount of unlabeled data increases. Furthermore, we find that combining them in a semi-supervised setup is highly synergistic: verifiable rewards help stabilize the CCR, while CCR improves the generalizability of verifiable rewards to related tasks. We also provide a comprehensive empirical comparison of various reward sources across multiple model architectures and dataset sizes. Our results show that this semi-supervised approach consistently enhances mathematical reasoning, yielding an absolute improvement of 5-10% on multiple reasoning tasks.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 40 references that are in the map. First posted on arXiv 3 Apr 2026.

Builds on

28 of 40 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty22 Jul
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 23

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  7. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  8. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  9. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  10. 2016Sequence-Level Knowledge Distillationcited by 761
  11. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  12. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
11 more
  1. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  2. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  3. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  4. 2025CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillationcited by 216
  5. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
  6. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Controlcited by 136
  7. 2019Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewardscited by 76
  8. 2025Fine-Tuning Discrete Diffusion Models with Policy Gradient Methodscited by 36
  9. 2020ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translationcited by 34
  10. 2018Learning Approximate Inference Networks for Structured Predictioncited by 13
  11. 2023Unlocking Anticipatory Text Generation: A Constrained Approach for Large Language Models Decodingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Rubric-based On-policy Distillationcited by 11
  2. 2026Distill Where the Student Goes: Teacher-Regularized RL for English-Evidence Cross-Lingual RAGcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement-aware Knowledge Distillation for LLM Reasoning
  2. 2025TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning
  3. 2026Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving
  4. 2025Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models
  5. 2025ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
  6. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
  7. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  8. 2025KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning