mlsanityData to 14 Oct 2026

Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step

arXiv 2306.14050 ↗First posted 24 Jun 2023cs.CLcited by 148

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

also in 1 more
Abstract (arXiv)

Chain-of-thought prompting (e.g., "Let's think step-by-step") primes large language models to verbalize rationalization for their predictions. While chain-of-thought can lead to dramatic performance gains, benefits appear to emerge only for sufficiently large models (beyond 50B parameters). We show that orders-of-magnitude smaller models (125M -- 1.3B parameters) can still benefit from chain-of-thought prompting. To achieve this, we introduce Symbolic Chain-of-Thought Distillation (SCoTD), a method to train a smaller student model on rationalizations sampled from a significantly larger teacher model. Experiments across several commonsense benchmarks show that: 1) SCoTD enhances the performance of the student model in both supervised and few-shot settings, and especially for challenge sets; 2) sampling many reasoning chains per instance from the teacher is paramount; and 3) after distillation, student chain-of-thoughts are judged by humans as comparable to the teacher, despite orders of magnitude fewer parameters. We test several hypotheses regarding what properties of chain-of-thought samples are important, e.g., diversity vs. teacher likelihood vs. open-endedness. We release our corpus of chain-of-thought samples and code.

Key facts. Cited by 148 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 42 of its 59 references that are in the map. First posted on arXiv 24 Jun 2023.

Builds on

42 of 59 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Large Language Models Are Reasoning Teachers20 Dec
  2. 2022Teaching Small Language Models to Reason16 Dec
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them17 Oct
  4. 2022Large Language Models are Zero-Shot Reasoners24 May
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  10. 2019Language Models as Knowledge Bases?cited by 1,936
  11. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  12. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
24 more
  1. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  2. 2022Large Language Models Can Self-Improvecited by 664
  3. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  4. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  5. 2021MetaICL: Learning to Learn In Contextcited by 454
  6. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  7. 2016Solving General Arithmetic Word Problemscited by 360
  8. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  9. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 202
  10. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  11. 2021Generating Datasets with Pretrained Language Modelscited by 193
  12. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 177
  13. 2016Generating Visual Explanationscited by 158
  14. 2020WT5?! Training Text-to-Text Models to Explain their Predictionscited by 148
  15. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
  16. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 124
  17. 2019Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Modelcited by 102
  18. 2021e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Taskscited by 84
  19. 2021When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Datacited by 75
  20. 2018Semantically-Aligned Equation Generation for Solving and Reasoning Math Word Problemscited by 49
  21. 2018QuaRel: A Dataset and Models for Answering Questions about Qualitative Relationshipscited by 46
  22. 2016Rationale-Augmented Convolutional Neural Networks for Text Classificationcited by 39
  23. 2020Towards Interpretable Natural Language Understanding with Explanations as Latent Variablescited by 30
  24. 2022I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitationcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 23; 2024: 58; 2025: 45; 2026: 22.

148 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  2. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  3. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  4. 2023Efficient Large Language Models: A Surveycited by 186
  5. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  6. 2025MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learningcited by 173
  7. 2024Distilling System 2 into System 1cited by 159
  8. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  9. 2024Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoningcited by 110
  10. 2025Agentic Large Language Models, a surveycited by 108
  11. 2024Large Language Models Meet NLP: A Surveycited by 103
  12. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
38 more
  1. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 69
  2. 2023Auto-Regressive Next-Token Predictors are Universal Learnerscited by 67
  3. 2024SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMscited by 67
  4. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  5. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 56
  6. 2024TopViewRS: Vision-Language Models as Top-View Spatial Reasonerscited by 49
  7. 2024Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMscited by 48
  8. 2024Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillationcited by 47
  9. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Modelscited by 45
  10. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  11. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  12. 2024Adaptive In-conversation Team Building for Language Model Agentscited by 39
  13. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 38
  14. 2025Bayesian Teaching Enables Probabilistic Reasoning in Large Language Modelscited by 37
  15. 2024EVOLvE: Evaluating and Optimizing LLMs For In-Context Explorationcited by 34
  16. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanismcited by 32
  17. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  18. 2024Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Modelscited by 30
  19. 2025AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxescited by 30
  20. 2024Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpuscited by 29
  21. 2023CoTFormer: A Chain-of-Thought Driven Architecture with Budget-Adaptive Computation Cost at Inferencecited by 27
  22. 2023LLM-Assisted Code Cleaning For Training Accurate Code Generatorscited by 27
  23. 2023Chain-of-Thought in Neural Code Generation: From and For Lightweight Language Modelscited by 27
  24. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 26
  25. 2024Dual-Space Knowledge Distillation for Large Language Modelscited by 25
  26. 2024To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contextscited by 25
  27. 2023Thinking Like an Expert:Multimodal Hypergraph-of-Thought (HoT) Reasoning to boost Foundation Modalscited by 24
  28. 2024Learning to Maximize Mutual Information for Chain-of-Thought Distillationcited by 23
  29. 2023Tailoring Self-Rationalizers with Multi-Reward Distillationcited by 21
  30. 2024Distilling Large Language Models for Text-Attributed Graph Learningcited by 20
  31. 2023Mixed Distillation Helps Smaller Language Model Better Reasoningcited by 19
  32. 2024Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQAcited by 19
  33. 2024BiLD: Bi-directional Logits Difference Loss for Large Language Model Distillationcited by 19
  34. 2024Divide-or-Conquer? Which Part Should You Distill Your LLM?cited by 17
  35. 2024Path-Consistency with Prefix Enhancement for Efficient Inference in LLMscited by 17
  36. 2024SelectLLM: Can LLMs Select Important Instructions to Annotate?cited by 16
  37. 2024Investigating Mysteries of CoT-Augmented Distillationcited by 16
  38. 2024CoMAT: Chain of Mathematically Annotated Thought Improves Mathematical Reasoningcited by 16

50 most cited of 148 All 148 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
  2. 2026Structural Chain-of-Thought: Reasoning with Executable Computational Graphs
  3. 2023Implicit Chain of Thought Reasoning via Knowledge Distillation
  4. 2026Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information
  5. 2023SCOTT: Self-Consistent Chain-of-Thought Distillation
  6. 2025Rethinking the Chain-of-Thought: The Roles of In-Context Learning and Pre-trained Priors
  7. 2023Contrastive Chain-of-Thought Prompting
  8. 2025SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought