Analyzing the Power of Chain of Thought through Memorization Capabilities
On the map: LLM reasoning → Prompting and chain-of-thought reasoning
Abstract (arXiv)
It has been shown that the chain of thought (CoT) can enhance the power of large language models (LLMs) to solve certain mathematical reasoning problems. However, the capacity of CoT is still not fully explored. As an important instance, the following basic question has not yet been answered: Does CoT expand the capability of transformers across all reasoning tasks? We demonstrate that reasoning with transformers is essentially a memorization problem for reasoning datasets. Thus, examining the power of CoT across all reasoning tasks amounts to analyzing the memorization capabilities of CoT transformers. In this paper, we give a complete description of the memorization capabilities of fixed-precision transformers with or without CoT and give a negative answer to the above-mentioned question. Precisely, we first give necessary and sufficient conditions for fixed-precision transformers with and without CoT to memorize a finite reasoning dataset and show that these two conditions do not imply each other. Then, we give lower and upper bounds for the number of parameters needed for transformers with or without CoT to memorize a finite reasoning dataset with N elements, which are overlineTheta(N) in all cases. This implies that there exist reasoning tasks for which CoT does not enhance the reasoning power of transformers, leading to a negative answer to the above-mentioned question. Finally, we give the first results on memorizing infinite reasoning datasets by CoT transformers and show that some simple infinite datasets cannot be memorized by transformers with or without CoT.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 47 references that are in the map. First posted on arXiv 3 Nov 2025.
Builds on
30 of 47 references.
Closest ancestors · 9
Same direction, newest first
- 2025The Curse of CoT: On the Limitations of Chain-of-Thought in In-Context Learning7 Apr
- 2024Chain-of-Probe: Examining the Necessity and Accuracy of CoT Step-by-Step23 Jun
- 2024On the Representational Capacity of Neural Language Models with Chain-of-Thought Reasoning20 Jun
- 2024Chain of Thoughtlessness? An Analysis of CoT in Planning8 May
- 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problems20 Feb
- 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective24 May
- 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan
Other references · 21
- 2017Attention Is All You Needcited by 60,361
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
- 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
- 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
- 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
- 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
- 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 311
- 2022The Parallelism Tradeoff: Limitations of Log-Precision Transformerscited by 193
- 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
- 2016Identity Matters in Deep Learningcited by 109
9 more
- 2023Tighter Bounds on the Expressivity of Transformer Encoderscited by 66
- 2023Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?cited by 47
- 2021On the Optimal Memorization Power of ReLU Neural Networkscited by 40
- 2023Memorization Capacity of Multi-Head Attention in Transformerscited by 40
- 2020Provable Memorization via Deep Neural Networks using Sub-linear Parameterscited by 34
- 2024How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMscited by 24
- 2022Why Robust Generalization in Deep Learning is Difficult: Perspective of Expressive Powercited by 23
- 2024Next-token prediction capacity: general upper bounds and a lower bound for transformerscited by 11
- 2024On the Optimal Memorization Capacity of Transformerscited by 10
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026On the Capability and Limitation of Hard Promptcited by 0
- 2026On the Capability and Limitation of Hard Prompt26 Sep 2026
Similar, not cited
Close in wording, no citation either way
- 2025The Kinetics of Reasoning: How Chain-of-Thought Shapes Learning in Transformers?
- 2025Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning
- 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models
- 2025Latent Chain-of-Thought? Decoding the Depth-Recurrent Transformer
- 2026Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbations
- 2025Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoning
- 2024The Impact of Reasoning Step Length on Large Language Models