mlsanityData to 14 Oct 2026

Function-Space Temporal Self-Distillation for Policy Optimization

arXiv 2610.19687 ↗First posted 14 Oct 2026cs.LG cs.AI cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in language models, but scalar outcomes provide coarse supervision over response trajectories. Self-distillation offers denser token-level guidance by comparing an unassisted policy with a feedback-informed self-teacher, yet the teacher evolves with the student, creating unstable targets and motivating additional regularization. We introduce Function-Space Policy Optimization (FSPO), a single-policy framework that combines current privileged adaptation with function-space temporal consolidation. FSPO stores sparse historical next-token distributions as fixed temporal targets, providing temporal separation without maintaining a persistent teacher model. It further introduces privileged disagreement correction to explicitly suppress locally overvalued actions, reliability-adaptive grounding that strengthens reference supervision when self-generated evidence repeatedly fails, and quality and coverage-aware temporal memory that retains verified behaviors and improves privileged demonstrations. Across reasoning and tool-use benchmarks, FSPO achieves the highest average accuracy under a 5-hour training budget, improving over the strongest baseline by approximately 4% points on average while reducing peak memory from about 70 GB to 56 GB on Olmo3-7B-Instruct.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 59 of its 66 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

59 of 66 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  3. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  4. 2025Process Reinforcement through Implicit Rewards3 Feb
  5. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  6. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  7. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  8. 2023Let's Verify Step by Step31 May
  9. 2017Proximal Policy Optimization Algorithms20 Jul
  10. 2017Thinking Fast and Slow with Deep Learning and Tree Search23 May

Other references · 49

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  7. 2015Trust Region Policy Optimizationcited by 2,580
  8. 2016Learning without Forgettingcited by 2,496
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  10. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  11. 2017Hindsight Experience Replaycited by 1,136
  12. 2018Experience Replay for Continual Learningcited by 1,095
37 more
  1. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  2. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  3. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
  4. 2016Sequence-Level Knowledge Distillationcited by 801
  5. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  6. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
  7. 2026Reinforcement Learning via Self-Distillationcited by 332
  8. 2024Titans: Learning to Memorize at Test Timecited by 328
  9. 2026Self-Distillation Enables Continual Learningcited by 247
  10. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  11. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 176
  12. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 148
  13. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 139
  14. 2022Learning by Distilling Contextcited by 138
  15. 2023Training Language Models with Language Feedback at Scalecited by 128
  16. 2025Competitive Programming with Large Reasoning Modelscited by 119
  17. 2020Continual Deep Learning by Functional Regularisation of Memorable Pastcited by 105
  18. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?cited by 103
  19. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 101
  20. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 96
  21. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 94
  22. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 86
  23. 2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 78
  24. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 62
  25. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 53
  26. 2022Prompt Injection: Parameterization of Fixed Inputscited by 36
  27. 2024Efficient Knowledge Injection in LLMs via Self-Distillationcited by 24
  28. 2025Language Models Can Learn from Verbal Feedback Without Scalar Rewardscited by 18
  29. 2025InfiniteICL: Breaking the Limit of Context Window Size via Long Short-term Memory Transformationcited by 17
  30. 2025SCAR: Shapley Credit Assignment for More Efficient RLHFcited by 16
  31. 2025Better Estimation of the Kullback--Leibler Divergence Between Language Modelscited by 11
  32. 2024Reinforcement Learning from Reflective Feedback (RLRF): Aligning and Improving LLMs via Fine-Grained Self-Reflectioncited by 8
  33. 2026Uncertainty-Aware Vision-Language Segmentation for Medical Imagingcited by 5
  34. 2026Efficient Text-Guided Convolutional Adapter for the Diffusion Modelcited by 4
  35. 2026FOCUS: Forcing In-Context Object Localization through Visual Support Constraints and Policy Optimizationcited by 3
  36. 2026UNITY: Attention Flow Networks for Adaptive Conditioning in Diffusioncited by 3
  37. 2026SALD: Self-Referenced Advantage Learning for Diffusion Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation
  2. 2026Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence
  3. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing
  4. 2026VASPO: Self-Distillation with Privileged Value Estimation
  5. 2026Self-Distilled Policy Gradient
  6. 2026On the Geometry of On-Policy Distillation
  7. 2026DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models
  8. 2026Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models