mlsanityData to 7 Oct 2026

Investigating Mysteries of CoT-Augmented Distillation

arXiv 2406.14511 ↗First posted 20 Jun 2024cs.CLcited by 15

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Eliciting "chain of thought" (CoT) rationales -- sequences of token that convey a "reasoning" process -- has been shown to consistently improve LLM performance on tasks like question answering. More recent efforts have shown that such rationales can also be used for model distillation: Including CoT sequences (elicited from a large "teacher" model) in addition to target labels when fine-tuning a small student model yields (often substantial) improvements. In this work we ask: Why and how does this additional training signal help in model distillation? We perform ablations to interrogate this, and report some potentially surprising results. Specifically: (1) Placing CoT sequences after labels (rather than before) realizes consistently better downstream performance -- this means that no student "reasoning" is necessary at test time to realize gains. (2) When rationales are appended in this way, they need not be coherent reasoning sequences to yield improvements; performance increases are robust to permutations of CoT tokens, for example. In fact, (3) a small number of key tokens are sufficient to achieve improvements equivalent to those observed when full rationales are used in model distillation.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 22 of its 29 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

22 of 29 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Preemptive Answer "Attacks" on Chain-of-Thought Reasoning31 May
  2. 2023Implicit Chain of Thought Reasoning via Knowledge Distillation2 Nov
  3. 2023DialCoT Meets PPO: Decomposing and Exploring Reasoning Paths in Smaller Language Models8 Oct
  4. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step24 Jun
  5. 2023SCOTT: Self-Consistent Chain-of-Thought Distillation3 May
  6. 2022Large Language Models Are Reasoning Teachers20 Dec
  7. 2022Distilling Reasoning Capabilities into Smaller Language Models1 Dec

Other references · 15

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  3. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  6. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  7. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  8. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  9. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  10. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 249
  11. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
  12. 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 116
3 more
  1. 2023Revisiting Relation Extraction in the era of Large Language Modelscited by 100
  2. 2021When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Datacited by 75
  3. 2018QuaRel: A Dataset and Models for Answering Questions about Qualitative Relationshipscited by 46

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 6; 2026: 4.

15 papers.

Sort:

By area:

  1. 2024What Makes Large Language Models Reason in (Multi-Turn) Code Generation?cited by 29
  2. 2025Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMscited by 25
  3. 2025Not All Errors Are Created Equal: ASCoT Addresses Late-Stage Fragility in Efficient LLM Reasoningcited by 11
  4. 2025Who Taught You That? Tracing Teachers in Model Distillationcited by 8
  5. 2025Mitigating Spurious Correlations Between Question and Answer via Chain-of-Thought Correctness Perception Distillationcited by 6
  6. 2024Boosting LLM Translation Skills without General Ability Loss via Rationale Distillationcited by 5
  7. 2024OrthoDoc: Multimodal Large Language Model for Assisting Diagnosis in Computed Tomographycited by 3
  8. 2025Kongzi: A Historical Large Language Model with Fact Enhancementcited by 2
  9. 2024How Performance Pressure Influences AI-Assisted Decision Makingcited by 1
  10. 2026CoDe-R: Refining Decompiler Output with LLMs via Rationale Guidance and Adaptive Inferencecited by 1
  11. 2024Sentiment Reasoning for Healthcarecited by 0
  12. 2025Circuit Distillationcited by 0
3 more
  1. 2026Internalized Reasoning for Long-Context Visual Document Understandingcited by 0
  2. 2026Characterize Then Distill: Mechanistic Reasoning in Large Output Spacescited by 0
  3. 2026TELLER: Dual-Path Iterative Preference Optimization for Table Entity Linkingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Efficient CoT Distillation: Self-Guided Rationale Selector for Better Performance with Fewer Rationales
  2. 2025Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
  3. 2026MI-Distillation: Selecting from Model-Interpolated Instruct-Reasoning Data Spectrum for Chain-of-Thought Distillation
  4. 2024Beyond Imitation: Learning Key Reasoning Steps from Dual Chain-of-Thoughts in Reasoning Distillation
  5. 2025The Quest for Efficient Reasoning: A Data-Centric Benchmark to CoT Distillation
  6. 2025Deconstructing Long Chain-of-Thought: A Structured Reasoning Optimization Framework for Long CoT Distillation
  7. 2025The Challenge of Teaching Reasoning to LLMs Without RL or Distillation
  8. 2025Effectiveness of Chain-of-Thought in Distilling Reasoning Capability from Large Language Models