mlsanityData to 7 Oct 2026

TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillation

arXiv 2503.04872 ↗First posted 6 Mar 2025cs.CL cs.AIcited by 12

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

The challenge of reducing the size of Large Language Models (LLMs) while maintaining their performance has gained significant attention. However, existing methods, such as model distillation and transfer learning, often fail to achieve high accuracy. To address this limitation, we introduce the Branch-Merge distillation approach, which enhances model compression through two phases: (1) the Branch Phase, where knowledge from a large teacher model is textitselectively distilled into specialized student models via domain-specific supervised fine-tuning (SFT); And (2) the Merge Phase, where these student models are merged to enable cross-domain knowledge transfer and improve generalization. We validate our distillation approach using DeepSeek-R1 as the teacher and DeepSeek-R1-Distill-Qwen-32B as the student. The resulting merged model, TinyR1-32B-Preview, outperforms its counterpart DeepSeek-R1-Distill-Qwen-32B across multiple benchmarks, including Mathematics (+5.5 points), Coding (+4.4 points) and Science (+2.9 points), while achieving near-equal performance to DeepSeek-R1 on AIME 2024. The Branch-Merge distillation approach provides a scalable solution for creating smaller, high-performing LLMs with reduced computational cost and time.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 46 of its 69 references that are in the map. First posted on arXiv 6 Mar 2025.

Builds on

46 of 69 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025s1: Simple test-time scaling31 Jan

Other references · 45

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  4. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  5. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  6. 2020Knowledge Distillation: A Surveycited by 1,578
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  9. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  10. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  12. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
33 more
  1. 2020Efficient Transformers: A Surveycited by 784
  2. 2024Self-Rewarding Language Modelscited by 654
  3. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  4. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  5. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  6. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  7. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  8. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  9. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  10. 2021A Survey of Data Augmentation Approaches for NLPcited by 408
  11. 2020What is being transferred in transfer learning?cited by 365
  12. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  13. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  14. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  15. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  16. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  17. 2022Patching open-vocabulary models by interpolating weightscited by 187
  18. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  19. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 130
  20. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 123
  21. 2022Fusing finetuned models for better pretrainingcited by 110
  22. 2021Learning Neural Network Subspacescited by 82
  23. 2022When Do Flat Minima Optimizers Work?cited by 78
  24. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
  25. 2024DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Samplingcited by 77
  26. 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
  27. 2019Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Netscited by 63
  28. 2024FuseChat: Knowledge Fusion of Chat Modelscited by 53
  29. 2024SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Modelscited by 38
  30. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  31. 2019AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learningcited by 22
  32. 2024Hummer: Towards Limited Competitive Preference Datasetcited by 10
  33. 2019A Deep Neural Network's Loss Surface Contains Every Low-dimensional Patterncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 4.

12 papers.

  1. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  2. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
  3. 2025Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Followingcited by 6
  4. 2026Skill-Aware Data Selection and Fine-Tuning for Data-Efficient Reasoning Distillationcited by 4
  5. 2026Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibilitycited by 4
  6. 2025Can Past Experience Accelerate LLM Reasoning?cited by 3
  7. 2025Evaluation is All You Need: Strategic Overclaiming of LLM Reasoning Capabilities Through Evaluation Designcited by 3
  8. 2025MASTER: Enhancing Large Language Model via Multi-Agent Simulated Teachingcited by 2
  9. 2025Won: Establishing Best Practices for Korean Financial NLPcited by 0
  10. 2025Composable Cross-prompt Essay Scoring by Merging Modelscited by 0
  11. 2026Beyond Parameter Arithmetic: Sparse Complementary Fusion for Distribution-Aware Model Mergingcited by 0
  12. 2026MedLoc-R1: Performance-Aware Curriculum Reward Scheduling for GRPO-Based Medical Visual Groundingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
  2. 2026Thinking at the Right Size: Amortized Distillation Across Post-Trained LLMs
  3. 2026WASD: Wasserstein-based Knowledge Distillation for Large Language Models
  4. 2024Self-Data Distillation for Recovering Quality in Pruned Large Language Models
  5. 2023Just CHOP: Embarrassingly Simple LLM Compression
  6. 2024Enhancing Knowledge Distillation of Large Language Models through Efficient Multi-Modal Distribution Alignment
  7. 2026Scaling Laws for Task-Specific LLM Distillation
  8. 2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models