mlsanityData to 7 Oct 2026

Implicit Chain of Thought Reasoning via Knowledge Distillation

arXiv 2311.01460 ↗First posted 2 Nov 2023cs.CL cs.AI cs.LGcited by 165

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

To augment language models with the ability to reason, researchers usually prompt or finetune them to produce chain of thought reasoning steps before producing the final answer. However, although people use natural language to reason effectively, it may be that LMs could reason more effectively with some intermediate computation that is not in natural language. In this work, we explore an alternative reasoning approach: instead of explicitly producing the chain of thought reasoning steps, we use the language model's internal hidden states to perform implicit reasoning. The implicit reasoning steps are distilled from a teacher model trained on explicit chain-of-thought reasoning, and instead of doing reasoning "horizontally" by producing intermediate words one-by-one, we distill it such that the reasoning happens "vertically" among the hidden states in different layers. We conduct experiments on a multi-digit multiplication task and a grade school math problem dataset and find that this approach enables solving tasks previously not solvable without explicit chain-of-thought, at a speed comparable to no chain-of-thought.

Key facts. Cited by 165 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 37 of its 47 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

37 of 47 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Models18 Aug
  2. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks22 Nov
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them17 Oct
  4. 2022Complexity-Based Prompting for Multi-Step Reasoning3 Oct
  5. 2022Large Language Models are Zero-Shot Reasoners24 May
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  9. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  12. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
18 more
  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2022Emergent Abilities of Large Language Modelscited by 2,483
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  7. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  8. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  9. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  10. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  11. 2016Sequence-Level Knowledge Distillationcited by 761
  12. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  13. 2017Non-Autoregressive Neural Machine Translationcited by 457
  14. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  15. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  16. 2019Lagging Inference Networks and Posterior Collapse in Variational Autoencoderscited by 124
  17. 2018Language GANs Falling Shortcited by 97
  18. 2022NaturalProver: Grounded Mathematical Proof Generation with Language Modelscited by 72

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 49; 2026: 89.

165 papers.

Sort:

By area:

  1. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  2. 2024Scaling up Masked Diffusion Models on Textcited by 195
  3. 2024Compressed Chain of Thought: Efficient Reasoning Through Dense Representationscited by 192
  4. 2024Distilling System 2 into System 1cited by 152
  5. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 140
  6. 2025Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokenscited by 120
  7. 2024InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoningcited by 117
  8. 2024C3oT: Generating Shorter Chain-of-Thought without Compromising Effectivenesscited by 117
  9. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  10. 2025Efficient Reasoning Models: A Surveycited by 93
  11. 2025Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoningcited by 87
  12. 2025Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chainscited by 81
38 more
  1. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 78
  2. 2025Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoningcited by 72
  3. 2025Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokenscited by 65
  4. 2024Can Language Models Learn to Skip Steps?cited by 59
  5. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 58
  6. 2025Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinkingcited by 56
  7. 2025Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Modelscited by 53
  8. 2024Transformers Provably Solve Parity Efficiently with Chain of Thoughtcited by 49
  9. 2025Reinforced Latent Reasoning for LLM-based Recommendationcited by 46
  10. 2025Implicit Reasoning in Large Language Models: A Comprehensive Surveycited by 42
  11. 2024MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Modelscited by 41
  12. 2025Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step Entropycited by 37
  13. 2025Bayesian Teaching Enables Probabilistic Reasoning in Large Language Modelscited by 36
  14. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 35
  15. 2024LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraintscited by 34
  16. 2025LLMs are Single-threaded Reasoners: Demystifying the Working Mechanism of Soft Thinkingcited by 34
  17. 2025LLM Pretraining with Continuous Conceptscited by 33
  18. 2024Break the Chain: Large Language Models Can be Shortcut Reasonerscited by 32
  19. 2025Continuous Chain of Thought Enables Parallel Exploration and Reasoningcited by 32
  20. 2025Hybrid Latent Reasoning via Reinforcement Learningcited by 30
  21. 2024Calibrating Reasoning in Language Models with Internal Consistencycited by 27
  22. 2025LaDiR: Latent Diffusion Enhances LLMs for Text Reasoningcited by 25
  23. 2025VeriThinker: Learning to Verify Makes Reasoning Model Efficientcited by 24
  24. 2025System-1.5 Reasoning: Traversal in Language and Latent Spaces with Dynamic Shortcutscited by 23
  25. 2025Do Latent Tokens Think? A Causal and Adversarial Analysis of Chain-of-Continuous-Thoughtcited by 20
  26. 2026Render-of-Thought: Rendering Textual Chain-of-Thought as Images for Visual Latent Reasoningcited by 19
  27. 2024Expediting and Elevating Large Language Model Reasoning via Hidden Chain-of-Thought Decodingcited by 18
  28. 2024Probabilistic Medical Predictions of Large Language Modelscited by 16
  29. 2024Do LLMs Really Think Step-by-step In Implicit Reasoning?cited by 16
  30. 2025Beyond the Last Answer: Your Reasoning Trace Uncovers More than You Thinkcited by 16
  31. 2024Investigating Mysteries of CoT-Augmented Distillationcited by 15
  32. 2025Skip-Thinking: Chunk-wise Chain-of-Thought Distillation Enable Smaller Language Models to Reason Better and Fastercited by 15
  33. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  34. 2025Consistency in Language Models: Current Landscape, Challenges, and Future Directionscited by 14
  35. 2025Can Aha Moments Be Fake? Towards Quantifying Decorative and True Thinking in Chain-of-Thoughtcited by 14
  36. 2024Language Models are Symbolic Learners in Arithmeticcited by 11
  37. 2026Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformerscited by 11
  38. 2025Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoningcited by 10

50 most cited of 165 All 165 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Teaching Small Language Models to Reason
  2. 2025KisMATH: Do LLMs Have Knowledge of Implicit Structures in Mathematical Reasoning?
  3. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step
  4. 2024CoMAT: Chain of Mathematically Annotated Thought Improves Mathematical Reasoning
  5. 2025Premise-Augmented Reasoning Chains Improve Error Identification in Math reasoning with LLMs
  6. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought
  7. 2025Deductive Logic in Language Models: Horizontal vs Vertical Reasoning
  8. 2025Efficient Reasoning via Chain of Unconscious Thought