Collaborative Reasoning Distillation via Cross-Feedback and Coherent Curation
On the map: LLM reasoning → Prompting and chain-of-thought reasoning also fits: Reasoning benchmarks and evaluation
Abstract (arXiv)
Reasoning capabilities are critical for advancing Large Language Models, yet current approaches either require massive computational budgets or struggle to effectively distill reasoning to smaller models. Standard distillation methods rely on outcome-based rewards, failing to distinguish between sound reasoning and lucky guesses. We propose Collaborative Reasoning Distillation (CRD), a framework that enhances reasoning in compact models through three innovations: (1) interactive cross-feedback where teachers iteratively critique each other's reasoning, (2) fine-grained step-wise quality assessment capturing logical validity independent of final answers, and (3) coherence-aware step stitching that synthesizes complementary strengths. Students are trained via Reasoning Quality Optimization (RQO) with budget constraints. Our model, CRD-4B, achieves 97.3% on MATH-500 and 70.3% on AIME'25, surpassing baselines while using only 50K training examples, up to 12 times smaller than the datasets of comparable models.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 44 of its 49 references that are in the map. First posted on arXiv 7 Oct 2026.
Builds on
44 of 49 references.
Closest ancestors · 6
Same direction, newest first
- 2025MiCoTA: Bridging the Learnability Gap with Intermediate CoT and Teacher Assistants2 Jul
- 2025Chain of Draft: Thinking Faster by Writing Less25 Feb
- 2024Learning from Committee: Reasoning Distillation from a Mixture of Teachers with Peer-Review4 Oct
- 2024Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillation7 Feb
- 2022Teaching Small Language Models to Reason16 Dec
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan
Other references · 38
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2023Let's Verify Step by Stepcited by 3,598
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
- 2024A Survey on LLM-as-a-Judgecited by 1,384
- 2025s1: Simple test-time scalingcited by 1,311
26 more
- 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
- 2019Contrastive Representation Distillationcited by 734
- 2025LIMO: Less is More for Reasoningcited by 488
- 2025Process Reinforcement through Implicit Rewardscited by 478
- 2022Self-critiquing models for assisting human evaluatorscited by 372
- 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
- 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 249
- 2021Does Knowledge Distillation Really Work?cited by 187
- 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 187
- 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 123
- 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
- 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 111
- 2024The rising costs of training frontier AI modelscited by 91
- 2024Process Reward Model with Q-Value Rankingscited by 78
- 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
- 2025Unlocking Efficient Long-to-Short LLM Reasoning with Model Mergingcited by 58
- 2025xVerify: Efficient Answer Verifier for Reasoning Model Evaluationscited by 53
- 2024MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Modelscited by 41
- 2025SCP-116K: A High-Quality Problem-Solution Dataset and a Generalized Pipeline for Automated Extraction in the Higher Education Science Domaincited by 30
- 2025EXAONE Deep: Reasoning Enhanced Language Modelscited by 20
- 2025Train Long, Think Short: Curriculum Learning for Efficient Reasoningcited by 20
- 2025Leash: Adaptive Length Penalty and Reward Shaping for Efficient Large Reasoning Modelcited by 13
- 2025A Comparison of DeepSeek and Other LLMscited by 10
- 2025Hermes 4 Technical Reportcited by 5
- 2025Demystifying Errors in LLM Reasoning Traces: An Empirical Study of Code Execution Simulationcited by 5
- 2025The Valley of Code Reasoning: Scaling Knowledge Distillation of Large Language Modelscited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026CORE: Collaborative Reasoning via Cross Teaching
- 2026Validity-Calibrated Reasoning Distillation
- 2025Learning from Diverse Reasoning Paths with Routing and Collaboration
- 2025Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
- 2025Not All Correct Answers Are Equal: Why Your Distillation Source Matters
- 2025Critique-Guided Distillation for Robust Reasoning via Refinement
- 2026Tailoring the Curriculum: Student-Centered Reasoning Distillation via Dynamic Data-Model Compatibility
- 2026GateKD: Confidence-Gated Closed-Loop Distillation for Robust Reasoning