mlsanityData to 7 Oct 2026

Learning to Maximize Mutual Information for Chain-of-Thought Distillation

arXiv 2403.03348 ↗First posted 5 Mar 2024cs.CL cs.AIcited by 23

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Knowledge distillation, the technique of transferring knowledge from large, complex models to smaller ones, marks a pivotal step towards efficient AI deployment. Distilling Step-by-Step~(DSS), a novel method utilizing chain-of-thought~(CoT) distillation, has demonstrated promise by imbuing smaller models with the superior reasoning capabilities of their larger counterparts. In DSS, the distilled model acquires the ability to generate rationales and predict labels concurrently through a multi-task learning framework. However, DSS overlooks the intrinsic relationship between the two training tasks, leading to ineffective integration of CoT knowledge with the task of label prediction. To this end, we investigate the mutual relationship of the two tasks from Information Bottleneck perspective and formulate it as maximizing the mutual information of the representation features of the two tasks. We propose a variational approach to solve this optimization problem using a learning-based method. Our experimental results across four datasets demonstrate that our method outperforms the state-of-the-art DSS. Our findings offer insightful guidance for future research on language model distillation as well as applications involving CoT. Codes are available at urlhttps://github.com/xinchen9/cotdistillationACL2024.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 9 of its 60 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

9 of 60 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Sci-CoT: Leveraging Large Language Models for Enhanced Knowledge Distillation in Small Models for Scientific QA9 Aug
  2. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step24 Jun

Other references · 7

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  6. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  7. 2018Universal Language Model Fine-tuning for Text Classificationcited by 50

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 14; 2026: 4.

23 papers.

Sort:

By area:

  1. 2024Distilling System 2 into System 1cited by 152
  2. 2025Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoningcited by 50
  3. 2025SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Paperscited by 42
  4. 2025Bayesian Teaching Enables Probabilistic Reasoning in Large Language Modelscited by 36
  5. 2025AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxescited by 28
  6. 2024Distilling Reasoning Ability from Large Language Models with Adaptive Thinkingcited by 13
  7. 2025Activation-Guided Consensus Merging for Large Language Modelscited by 12
  8. 2024Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Datacited by 9
  9. 2024QCRD: Quality-guided Contrastive Rationale Distillation for Large Language Modelscited by 8
  10. 2024Data-Centric Human Preference with Rationales for Direct Preference Alignmentcited by 4
  11. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  12. 2025Predicting Through Generation: Why Generation Is Better for Predictioncited by 2
11 more
  1. 2025Kongzi: A Historical Large Language Model with Fact Enhancementcited by 2
  2. 2025Honey, I Shrunk the Language Model: Impact of Knowledge Distillation Methods on Performance and Explainabilitycited by 2
  3. 2026Pedagogically-Inspired Data Synthesis for Language Model Knowledge Distillationcited by 2
  4. 2026Shorthand for Thought: Compressing LLM Reasoning via Entropy-Guided Supertokenscited by 2
  5. 2025FALCON: Fine-grained Activation Manipulation by Contrastive Orthogonal Unalignment for Large Language Modelcited by 1
  6. 2025FedCoT: Communication-Efficient Federated Reasoning Enhancement for Large Language Modelscited by 1
  7. 2025What's on My Network? Using Large Language Models to Identify Real-World IoT Devices at Scalecited by 1
  8. 2025Breaking Memorization Barriers in LLM Code Fine-Tuning via Information Bottleneck for Improved Generalizationcited by 1
  9. 2026"The Whole Is Greater Than the Sum of Its Parts": A Compatibility-Aware Multi-Teacher CoT Distillation Frameworkcited by 1
  10. 2025GEM: Generative Entropy-Guided Preference Modeling for Few-shot Alignment of LLMscited by 0
  11. 2026Know Thyself, Teach Thyself: Internal Information Flow for Selective Self-Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MI-Distillation: Selecting from Model-Interpolated Instruct-Reasoning Data Spectrum for Chain-of-Thought Distillation
  2. 2026Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information
  3. 2022Distilling Reasoning Capabilities into Smaller Language Models
  4. 2025Deconstructing Long Chain-of-Thought: A Structured Reasoning Optimization Framework for Long CoT Distillation
  5. 2025Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
  6. 2023Mixed Distillation Helps Smaller Language Model Better Reasoning
  7. 2024Keypoint-based Progressive Chain-of-Thought Distillation for LLMs
  8. 2023SCOTT: Self-Consistent Chain-of-Thought Distillation