mlsanityData to 7 Oct 2026

SCOTT: Self-Consistent Chain-of-Thought Distillation

arXiv 2305.01879 ↗First posted 3 May 2023cs.CLcited by 103

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Large language models (LMs) beyond a certain scale, demonstrate the emergent capability of generating free-text rationales for their predictions via chain-of-thought (CoT) prompting. While CoT can yield dramatically improved performance, such gains are only observed for sufficiently large LMs. Even more concerning, there is little guarantee that the generated rationales are consistent with LM's predictions or faithfully justify the decisions. In this work, we propose a faithful knowledge distillation method to learn a small, self-consistent CoT model from a teacher model that is orders of magnitude larger. To form better supervision, we elicit rationales supporting the gold answers from a large LM (teacher) by contrastive decoding, which encourages the teacher to generate tokens that become more plausible only when the answer is considered. To ensure faithful distillation, we use the teacher-generated rationales to learn a student LM with a counterfactual reasoning objective, which prevents the student from ignoring the rationales to make inconsistent predictions. Experiments show that, while yielding comparable end-task performance, our method can generate CoT rationales that are more faithful than baselines do. Further analysis suggests that such a model respects the rationales more when making decisions; thus, we can improve its performance more by refining its rationales.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 40 references that are in the map. First posted on arXiv 3 May 2023.

Builds on

24 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 23

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2022Emergent Abilities of Large Language Modelscited by 2,483
  5. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  6. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  7. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  8. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  9. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  10. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  11. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  12. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
11 more
  1. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  2. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  3. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  4. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 248
  5. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
  6. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 123
  7. 2020Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?cited by 95
  8. 2021CREAK: A Dataset for Commonsense Reasoning over Entity Knowledgecited by 88
  9. 2022PINTO: Faithful Language Reasoning Using Prompt-Generated Rationalescited by 56
  10. 2022Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Modelcited by 20
  11. 2022Investigating the Benefits of Free-Form Rationalescited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 46; 2025: 22; 2026: 19.

103 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 329
  3. 2023A Survey on Model Compression for Large Language Modelscited by 263
  4. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  5. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  6. 2024Distilling System 2 into System 1cited by 152
  7. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  8. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 145
  9. 2025Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketchingcited by 116
  10. 2024Large Language Models Can Learn Temporal Reasoningcited by 110
  11. 2024Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoningcited by 108
  12. 2024Large Language Models Meet NLP: A Surveycited by 102
38 more
  1. 2024Are self-explanations from Large Language Models faithful?cited by 101
  2. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  3. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  4. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 65
  5. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  6. 2024Building Guardrails for Large Language Modelscited by 61
  7. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  8. 2024A Survey on Transformer Compressioncited by 44
  9. 2023Dialogue Chain-of-Thought Distillation for Commonsense-aware Conversational Agentscited by 41
  10. 2024Dualformer: Controllable Fast and Slow Thinking by Learning with Randomized Reasoning Tracescited by 40
  11. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 36
  12. 2025Bayesian Teaching Enables Probabilistic Reasoning in Large Language Modelscited by 36
  13. 2024EVOLvE: Evaluating and Optimizing LLMs For In-Context Explorationcited by 33
  14. 2024Break the Chain: Large Language Models Can be Shortcut Reasonerscited by 32
  15. 2023HARE: Explainable Hate Speech Detection with Step-by-Step Reasoningcited by 30
  16. 2024Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challengescited by 30
  17. 2024Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Modelscited by 30
  18. 2023PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuningcited by 29
  19. 2024Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpuscited by 29
  20. 2025AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxescited by 28
  21. 2024Discerning and Resolving Knowledge Conflicts through Adaptive Decoding with Contextual Information-Entropy Constraintcited by 27
  22. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  23. 2024DDK: Distilling Domain Knowledge for Efficient Large Language Modelscited by 25
  24. 2023GrammarGPT: Exploring Open-Source LLMs for Native Chinese Grammatical Error Correction with Supervised Fine-Tuningcited by 23
  25. 2024Faster and Lighter LLMs: A Survey on Current Challenges and Way Forwardcited by 20
  26. 2024Divide-or-Conquer? Which Part Should You Distill Your LLM?cited by 17
  27. 2023Crystal: Introspective Reasoners Reinforced with Self-Feedbackcited by 15
  28. 2023What Makes it Ok to Set a Fire? Iterative Self-distillation of Contexts and Rationales for Disambiguating Defeasible Social and Moral Situationscited by 15
  29. 2024Optimizing Language Model's Reasoning Abilities with Weak Supervisioncited by 15
  30. 2024Investigating Mysteries of CoT-Augmented Distillationcited by 15
  31. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 15
  32. 2024Distilling Reasoning Ability from Large Language Models with Adaptive Thinkingcited by 13
  33. 2024Beyond Imitation: Learning Key Reasoning Steps from Dual Chain-of-Thoughts in Reasoning Distillationcited by 13
  34. 2024Distilling Text Style Transfer With Self-Explanation From LLMscited by 12
  35. 2024LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Modelscited by 12
  36. 2024ELAD: Explanation-Guided Large Language Models Active Distillationcited by 11
  37. 2025Structured Agent Distillation for Large Language Modelcited by 11
  38. 2023Few-Shot Data Synthesis for Open Domain Multi-Hop Question Answeringcited by 9

50 most cited of 103 All 103 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models
  2. 2025Deconstructing Long Chain-of-Thought: A Structured Reasoning Optimization Framework for Long CoT Distillation
  3. 2026Masked Distillation: Internalizing the Chain-of-Thought in Language Models
  4. 2025Skip-Thinking: Chunk-wise Chain-of-Thought Distillation Enable Smaller Language Models to Reason Better and Faster
  5. 2026MI-Distillation: Selecting from Model-Interpolated Instruct-Reasoning Data Spectrum for Chain-of-Thought Distillation
  6. 2026Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information
  7. 2025Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
  8. 2024On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models