mlsanityData to 7 Oct 2026

MCC-KD: Multi-CoT Consistent Knowledge Distillation

arXiv 2310.14747 ↗First posted 23 Oct 2023cs.CLcited by 26

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Large language models (LLMs) have showcased remarkable capabilities in complex reasoning through chain of thought (CoT) prompting. Recently, there has been a growing interest in transferring these reasoning abilities from LLMs to smaller models. However, achieving both the diversity and consistency in rationales presents a challenge. In this paper, we focus on enhancing these two aspects and propose Multi-CoT Consistent Knowledge Distillation (MCC-KD) to efficiently distill the reasoning capabilities. In MCC-KD, we generate multiple rationales for each question and enforce consistency among the corresponding predictions by minimizing the bidirectional KL-divergence between the answer distributions. We investigate the effectiveness of MCC-KD with different model architectures (LLaMA/FlanT5) and various model scales (3B/7B/11B/13B) on both mathematical reasoning and commonsense reasoning benchmarks. The empirical results not only confirm MCC-KD's superior performance on in-distribution datasets but also highlight its robust generalization ability on out-of-distribution datasets.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 36 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

29 of 36 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Large Language Models Are Reasoning Teachers20 Dec
  2. 2022Teaching Small Language Models to Reason16 Dec
  3. 2022Large Language Models are Zero-Shot Reasoners24 May
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar

Other references · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
13 more
  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  2. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  3. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  4. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  5. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  6. 2016Solving General Arithmetic Word Problemscited by 353
  7. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  8. 2023The False Promise of Imitating Proprietary LLMscited by 254
  9. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 249
  10. 2021When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Datacited by 75
  11. 2022PINTO: Faithful Language Reasoning Using Prompt-Generated Rationalescited by 56
  12. 2021Distilling Linguistic Context for Language Model Compressioncited by 33
  13. 2021Refining Language Models with Compositional Explanationscited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 9; 2025: 8; 2026: 8.

26 papers.

Sort:

By area:

  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  2. 2024A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthinesscited by 125
  3. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  4. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  5. 2024Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillationcited by 46
  6. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  7. 2024Learning from Committee: Reasoning Distillation from a Mixture of Teachers with Peer-Reviewcited by 14
  8. 2024Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarioscited by 12
  9. 2025Learning from Diverse Reasoning Paths with Routing and Collaborationcited by 12
  10. 2025TwT: Thinking without Tokens by Habitual Reasoning Distillation with Multi-Teachers' Guidancecited by 11
  11. 2025Understanding LLM Scientific Reasoning through Promptings and Model's Explanation on the Answerscited by 10
  12. 2025Mitigating Spurious Correlations Between Question and Answer via Chain-of-Thought Correctness Perception Distillationcited by 6
14 more
  1. 2025Towards Efficient CoT Distillation: Self-Guided Rationale Selector for Better Performance with Fewer Rationalescited by 6
  2. 2026Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillationcited by 5
  3. 2025Advantage-Guided Distillation for Preference Alignment in Small Language Modelscited by 4
  4. 2025Self-Enhanced Reasoning Training: Activating Latent Reasoning in Small Models for Enhanced Reasoning Distillationcited by 3
  5. 2026Student-in-the-Loop Chain-of-Thought Distillation via Generation-Time Selectioncited by 3
  6. 2024AS-ES Learning: Towards Efficient CoT Learning in Small Modelscited by 1
  7. 2026MIND: From Passive Mimicry to Active Reasoning through Capability-Aware Multi-Perspective CoT Distillationcited by 1
  8. 2026"The Whole Is Greater Than the Sum of Its Parts": A Compatibility-Aware Multi-Teacher CoT Distillation Frameworkcited by 1
  9. 2026Aligning Thoughts with Answers: Probability Rewards to Tame Thinking Driftcited by 1
  10. 2024Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Modelscited by 0
  11. 2025ReaLM: Reflection-Enhanced Autonomous Reasoning with Small Language Modelscited by 0
  12. 2026Structural Rationale Distillation via Reasoning Space Compressioncited by 0
  13. 2026Breaking the Reward Barrier: Accelerating Tree-of-Thought Reasoning via Speculative Explorationcited by 0
  14. 2026ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Effectiveness of Chain-of-Thought in Distilling Reasoning Capability from Large Language Models
  2. 2024Mentor-KD: Making Small Language Models Better Multi-step Reasoners
  3. 2023Sci-CoT: Leveraging Large Language Models for Enhanced Knowledge Distillation in Small Models for Scientific QA
  4. 2025Deconstructing Long Chain-of-Thought: A Structured Reasoning Optimization Framework for Long CoT Distillation
  5. 2022Distilling Reasoning Capabilities into Smaller Language Models
  6. 2024Improving Mathematical Reasoning Capabilities of Small Language Models via Feedback-Driven Distillation
  7. 2024Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation
  8. 2023Knowledge-Driven CoT: Exploring Faithful Reasoning in LLMs for Knowledge-intensive Question Answering