mlsanityData to 14 Oct 2026

Let's Think Dot by Dot: Hidden Computation in Transformer Language Models

arXiv 2404.15758 ↗First posted 24 Apr 2024cs.CL cs.AIcited by 191

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Chain-of-thought responses from language models improve performance across most benchmarks. However, it remains unclear to what extent these performance gains can be attributed to human-like task decomposition or simply the greater computation that additional tokens allow. We show that transformers can use meaningless filler tokens (e.g., '......') in place of a chain of thought to solve two hard algorithmic tasks they could not solve when responding without intermediate tokens. However, we find empirically that learning to use filler tokens is difficult and requires specific, dense supervision to converge. We also provide a theoretical characterization of the class of problems where filler tokens are useful in terms of the quantifier depth of a first-order formula. For problems satisfying this characterization, chain-of-thought tokens need not provide information about the intermediate computational steps involved in multi-token computations. In summary, our results show that additional tokens can provide computational benefits independent of token choice. The fact that intermediate tokens can act as filler tokens raises concerns about large language models engaging in unauditable, hidden computations that are increasingly detached from the observed chain-of-thought tokens.

Key facts. Cited by 191 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 14 of its 19 references that are in the map. First posted on arXiv 24 Apr 2024.

Builds on

14 of 19 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Measuring Faithfulness in Chain-of-Thought Reasoning17 Jul
  2. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective24 May
  3. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting7 May
  4. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them17 Oct
  5. 2022A Logic for Expressing Log-Precision Transformers6 Oct
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 8

  1. 2019Designing and Interpreting Probes with Control Taskscited by 640
  2. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
  3. 2023Think before you speak: Training Language Models With Pause Tokenscited by 264
  4. 2022The Parallelism Tradeoff: Limitations of Log-Precision Transformerscited by 193
  5. 2023Representational Strengths and Limitations of Transformerscited by 133
  6. 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 121
  7. 2024Why are Sensitive Functions Hard for Transformers?cited by 74
  8. 2024Do language models plan ahead for future tokens?cited by 49

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 28; 2025: 80; 2026: 83.

191 papers.

Sort:

By area:

  1. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  2. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 464
  3. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  4. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 290
  5. 2025CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillationcited by 225
  6. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  7. 2024Compressed Chain of Thought: Efficient Reasoning Through Dense Representationscited by 201
  8. 2025Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMscited by 163
  9. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 133
  10. 2025Efficient Reasoning Models: A Surveycited by 95
  11. 2025Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoningcited by 91
  12. 2025When Chain of Thought is Necessary, Language Models Struggle to Evade Monitorscited by 88
38 more
  1. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 78
  2. 2025Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thoughtcited by 77
  3. 2024Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Modelscited by 68
  4. 2024Understanding Transformer Reasoning Capabilities via Graph Algorithmscited by 67
  5. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 65
  6. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 61
  7. 2025Training Strategies for Efficient Embodied Reasoningcited by 55
  8. 2025A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformerscited by 53
  9. 2025Measuring Chain of Thought Faithfulness by Unlearning Reasoning Stepscited by 47
  10. 2024Thinking LLMs: General Instruction Following with Thought Generationcited by 45
  11. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  12. 2026The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlookcited by 43
  13. 2025Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokenscited by 41
  14. 2025Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!cited by 37
  15. 2025Soft Tokens, Hard Truthscited by 35
  16. 2025Enabling Agents to Communicate Entirely in Latent Spacecited by 34
  17. 2024Break the Chain: Large Language Models Can be Shortcut Reasonerscited by 32
  18. 2025Hybrid Latent Reasoning via Reinforcement Learningcited by 32
  19. 2025Large language models can learn and generalize steganographic chain-of-thought under process supervisioncited by 32
  20. 2024Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machinacited by 29
  21. 2025LaDiR: Latent Diffusion Enhances LLMs for Text Reasoningcited by 29
  22. 2025System-1.5 Reasoning: Traversal in Language and Latent Spaces with Dynamic Shortcutscited by 27
  23. 2024From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyondcited by 24
  24. 2025Knowing Before Saying: LLM Representations Encode Information About Chain-of-Thought Success Before Completioncited by 23
  25. 2025Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasonerscited by 22
  26. 2025SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMscited by 22
  27. 2025M1: Towards Scalable Test-Time Compute with Mamba Reasoning Modelscited by 21
  28. 2024Rational Metareasoning for Large Language Modelscited by 20
  29. 2025Exact Expressive Power of Transformers with Paddingcited by 19
  30. 2024Planning in a recurrent neural network that plays Sokobancited by 18
  31. 2025Towards Concise and Adaptive Thinking in Large Reasoning Models: A Surveycited by 18
  32. 2026HY-Embodied-0.5: Embodied Foundation Models for Real-World Agentscited by 18
  33. 2024Lessons from Studying Two-Hop Latent Reasoningcited by 17
  34. 2024Autonomous Agents for Collaborative Task under Information Asymmetrycited by 16
  35. 2025Multi-Token Prediction Needs Registerscited by 16
  36. 2025How do Transformers Learn Implicit Reasoning?cited by 16
  37. 2025PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Spacecited by 16
  38. 2025Emergence of Superposition: Unveiling the Training Dynamics of Chain of Continuous Thoughtcited by 16

50 most cited of 191 All 191 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023The Expressive Power of Transformers with Chain of Thought
  2. 2024Understanding Hidden Computations in Chain-of-Thought Reasoning
  3. 2026Adaptive Loops and Memory in Transformers: Think Harder or Know More?
  4. 2026Sparse Token Routing in Efficient Transformers
  5. 2026Barriers to Universal Reasoning With Transformers (And How to Overcome Them)
  6. 2022Transformers Learn Shortcuts to Automata
  7. 2026Protoreasoning in Tiny Transformers
  8. 2024Do Efficient Transformers Really Save Computation?