mlsanityData to 14 Oct 2026

Distilling Long-CoT Reasoning through Collaborative Step-wise Multi-Teacher Decoding

arXiv 2605.02290 ↗First posted 4 May 2026cs.AIcited by 0

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Distilling large reasoning models is essential for making Long-CoT reasoning practical, as full-scale inference remains computationally prohibitive. Existing curation-based approaches select complete reasoning traces post-hoc, overlooking collaboration among heterogeneous teachers and lacking dynamic exploration, which leads to redundant sampling and missed complementary reasoning. We introduce CoRD, a collaborative multi-teacher decoding framework that performs step-wise reasoning synthesis guided by predictive perplexity-based scoring and beam search. This enables heterogeneous LRMs to jointly construct coherent reasoning trajectories while efficiently preserving diverse, high-potential hypotheses. Experiments show that CoRD produces higher-quality reasoning data and achieves near teacher-level student performance with fewer, structured supervision signals, without substantial efficiency overhead. CoRD further generalizes well to out-of-domain and open-ended settings. The dataset and model are available at hrefhttps://github.com/DISL-Lab/CoRDhttps://github.com/DISL-Lab/CoRD.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 41 references that are in the map. First posted on arXiv 4 May 2026.

Builds on

33 of 41 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Tasks2 Jul
  2. 2025Marco-o1 v2: Towards Widening The Distillation Bottleneck for Reasoning Models3 Mar
  3. 2025Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning?26 Feb
  4. 2025Small Models Struggle to Learn from Strong Reasoners17 Feb
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 28

  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  4. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  5. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  6. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  7. 2016Sequence-Level Knowledge Distillationcited by 801
  8. 2025LIMO: Less is More for Reasoningcited by 498
  9. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  10. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 408
  11. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 326
  12. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 295
16 more
  1. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  2. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  3. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 175
  4. 2025Phi-4-reasoning Technical Reportcited by 151
  5. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 123
  6. 2025Efficient Test-Time Scaling via Self-Calibrationcited by 75
  7. 2024Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoningcited by 40
  8. 2024Learning to Summarize from LLM-generated Feedbackcited by 24
  9. 2025THINK-Bench: Evaluating Thinking Efficiency and Chain-of-Thought Quality of Large Reasoning Modelscited by 19
  10. 2024Enhancing Mathematical Reasoning in LLMs by Stepwise Correctioncited by 16
  11. 2024Enhancing LLM Reasoning with Multi-Path Collaborative Reactive and Reflection agentscited by 15
  12. 2025Why Distillation can Outperform Zero-RL: The Role of Flexible Reasoningcited by 15
  13. 2025Reinforcement Learning vs. Distillation: Understanding Accuracy and Capability in LLM Reasoningcited by 14
  14. 2025Thinking Preference Optimizationcited by 8
  15. 2025Communication is All You Need: Persuasion Dataset Construction via Multi-LLM Communicationcited by 5
  16. 2025ReFeed: Multi-dimensional Summarization Refinement with Reflective Reasoning on Feedbackcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026"The Whole Is Greater Than the Sum of Its Parts": A Compatibility-Aware Multi-Teacher CoT Distillation Framework
  2. 2026Collaborative Reasoning Distillation via Cross-Feedback and Coherent Curation
  3. 2025Merge-of-Thought Distillation
  4. 2026MI-Distillation: Selecting from Model-Interpolated Instruct-Reasoning Data Spectrum for Chain-of-Thought Distillation
  5. 2025Enhancing Long-Chain Reasoning Distillation through Error-Aware Self-Reflection
  6. 2026How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Data
  7. 2025The Signal is in the Steps: Local Scoring for Reasoning Data Selection
  8. 2022Large Language Models Are Reasoning Teachers