# How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Data

arXiv: https://arxiv.org/abs/2604.14164 · HTML: https://mlsanity.com/paper/2604.14164/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 21 of its 50 references that are in the map.
- First posted on arXiv 23 Mar 2026.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)
- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 21 of 21)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [OpenThoughts: Data Recipes for Reasoning Models](https://mlsanity.com/paper/2506.04178/) (2025)
- [DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning](https://mlsanity.com/paper/2504.11456/) (2025)
- [RL's Razor: Why Online Reinforcement Learning Forgets Less](https://mlsanity.com/paper/2509.04259/) (2025)
- [OpenCodeReasoning: Advancing Data Distillation for Competitive Coding](https://mlsanity.com/paper/2504.01943/) (2025)
- [Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting](https://mlsanity.com/paper/2510.18874/) (2025)
- [SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines](https://mlsanity.com/paper/2502.14739/) (2025)
- [AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy](https://mlsanity.com/paper/2506.13284/) (2025)
- [LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?](https://mlsanity.com/paper/2506.11928/) (2025)
- [Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning](https://mlsanity.com/paper/2506.02867/) (2025)
- [OJBench: A Competition Level Code Benchmark For Large Language Models](https://mlsanity.com/paper/2506.16395/) (2025)
- [A Survey of Neural Code Intelligence: Paradigms, Advances and Beyond](https://mlsanity.com/paper/2403.14734/) (2024)
- [AdaSwitch: Balancing Exploration and Guidance in Knowledge Distillation via Adaptive Switching](https://mlsanity.com/paper/2510.07842/) (2025)
- [StyleBench: Evaluating thinking styles in Large Language Models](https://mlsanity.com/paper/2509.20868/) (2025)
- [Your thoughts tell who you are: Characterize the reasoning patterns of LRMs](https://mlsanity.com/paper/2509.24147/) (2025)
- [Pipelined Decoder for Efficient Context-Aware Text Generation](https://mlsanity.com/paper/2506.23431/) (2025)

## Built on by (most cited first, 8 of 8)

- [SocraticPO: Policy Optimization via Interactive Guidance](https://mlsanity.com/paper/2606.09887/) (2026)
- [When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff](https://mlsanity.com/paper/2606.09932/) (2026)
- [Execution-First Synthetic Tool-Use Trace Generation for LLM Agents](https://mlsanity.com/paper/2607.29175/) (2026)
- [Shared SFT Lessons Across Alignment, Model Organisms, and Toy Models](https://mlsanity.com/paper/2607.26173/) (2026)
- [Learning to Adapt SFT Data for Better Reasoning Generalization](https://mlsanity.com/paper/2605.26924/) (2026)
- [Agentic Instruction Data Selection: Let DataMaster Interpret Your Intent](https://mlsanity.com/paper/2608.10579/) (2026)
- [Reason in Style: Discovering and Controlling Style in Language Models](https://mlsanity.com/paper/2610.00724/) (2026)
- [Continually Post-training Small Models via Synthetic Self-distillation](https://mlsanity.com/paper/2610.13113/) (2026)

## Abstract (arXiv)

A widely adopted strategy for model enhancement is to use synthetic data generated by a stronger model for supervised fine-tuning (SFT). However, for emerging reasoning models like Qwen3-8B, this approach often fails to improve reasoning capabilities and can even lead to a substantial drop in performance. In this work, we identify substantial stylistic divergence between teacher generated data and the distribution of student as a major factor impacting SFT. To bridge this gap, we propose a Teacher-Student Cooperation Data Synthesis framework (TESSY), which interleaves teacher and student models to alternately generate style and non-style tokens. Consequently, TESSY produces synthetic sequences that inherit the advanced reasoning capabilities of the teacher while maintaining stylistic consistency with the distribution of the student. In experiments on code generation using GPT-OSS-120B as the teacher, fine-tuning Qwen3-8B on teacher-generated data leads to performance drops of 3.25% on LiveCodeBench-Pro and 10.02% on OJBench, whereas TESSY achieves improvements of 11.25% and 6.68%.