RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer
On the map: LLM reasoning → Prompting and chain-of-thought reasoning
also in 1 more
Abstract (arXiv)
Multilingual reasoning transfer is crucial for extending reasoning capabilities of large language models (LLMs) beyond high-resource languages. On-policy self-distillation (OPSD) and its variants have emerged as a promising paradigm, providing dense token-level supervision on student-generated rollouts, yet their objectives do not explicitly prioritize reasoning signals most critical to cross-lingual transfer. We characterize that target-language reasoning comprises the generation of both surface text and reasoning pivots, which are decisions that advance or redirect the reasoning process and shape subsequent inference. This motivates concentrating privileged distillation around such pivots. We therefore propose RP-OPSD, Reasoning-Pivot-guided On-Policy Self-Distillation, using the distributional shift between matched teacher views with and without an English reference solution as an operational proxy to guide privileged distillation and reference anchoring. Experiments on mathematical reasoning benchmarks covering 17 languages and multiple difficulty levels show that our method outperforms strong multilingual reasoning baselines and OPSD variants. Further analysis reveals that RP-OPSD concentrates privileged distillation on reasoning-control and problem-condistioned state-update tokens, while downweighting it for tokens that mainly support surface realization. Our code is available at https://github.com/NJUNLP/RP-OPSD.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 45 references that are in the map. First posted on arXiv 6 Aug 2026.
Builds on
36 of 45 references.
Closest ancestors · 9
Same direction, newest first
- 2026Skill-Conditioned Gated Self-Distillation for LLM Reasoning27 May
- 2026ROSD: Reflective On-Policy Self-Distillation for Language Model Reasoning across Domains27 May
- 2026Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning13 May
- 2026Crosslingual On-Policy Self-Distillation for Multilingual Reasoning10 May
- 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models26 Jan
- 2025Long Chain-of-Thought Reasoning Across Languages20 Aug
- 2025Thought Anchors: Which LLM Reasoning Steps Matter?23 Jun
- 2023Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages23 Oct
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan
Other references · 27
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025Qwen3 Technical Reportcited by 8,820
- 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
- 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
- 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
- 2026Reinforcement Learning via Self-Distillationcited by 332
- 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 304
- 2026On-Policy Context Distillation for Language Modelscited by 186
- 2026TIP: Token Importance in On-Policy Distillationcited by 104
- 2024MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimizationcited by 69
- 2025PolyMath: Evaluating Mathematical Reasoning in Multilingual Contextscited by 56
15 more
- 2026Stable On-Policy Distillation through Adaptive Target Reformulationcited by 53
- 2026Fast and Effective On-policy Distillation from Reasoning Prefixescited by 53
- 2025Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoningcited by 51
- 2026Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillationcited by 51
- 2025Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Mathcited by 44
- 2024IrokoBench: A New Benchmark for African Languages in the Age of Large Language Modelscited by 43
- 2024Question Translation Training for Better Multilingual Reasoningcited by 42
- 2025Exploiting Tree Structure for Credit Assignment in RL Training of LLMscited by 27
- 2025Language Matters: How Do Multilingual Input and Reasoning Paths Affect Large Reasoning Models?cited by 20
- 2026STEPS: Selective On-Policy Self-Distillation for Reasoningcited by 16
- 2023Empowering Multi-step Reasoning across Languages via Tree-of-Thoughtscited by 8
- 2025How Does Alignment Enhance LLMs' Multilingual Capabilities? A Language Neurons Perspectivecited by 8
- 2026TAPO: Translation Augmented Policy Optimization for Multilingual Mathematical Reasoningcited by 6
- 2026R3S: Refining and Recovering Reinforcement Signals for Multilingual Understanding and Reasoningcited by 5
- 2026Efficient Multilingual Reasoning Transfer via Progressive Code-Switchingcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Diagnosing On-Policy Self-Distillation for Reasoning Language Models
- 2026A Brief Overview: On-Policy Self-Distillation In Large Language Models
- 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
- 2025Parallel Scaling Law: Unveiling Reasoning Generalization through A Cross-Linguistic Perspective
- 2026Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance
- 2026Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models
- 2026Purified OPSD: On-Policy Self-Distillation Without Losing How to Think
- 2026Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation