# Rethinking Thinking Tokens: Understanding Why They Underperform in Practice

arXiv: https://arxiv.org/abs/2411.11371 · HTML: https://mlsanity.com/paper/2411.11371/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 5 of its 8 references that are in the map.
- First posted on arXiv 18 Nov 2024.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 5 of 5)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective](https://mlsanity.com/paper/2305.15408/) (2023)
- [Thinking Tokens for Language Modeling](https://mlsanity.com/paper/2405.08644/) (2024)
- [ThoughtSource: A central hub for large language model reasoning data](https://mlsanity.com/paper/2301.11596/) (2023)

## Built on by (most cited first, 2 of 2)

- [Enhancing Latent Computation in Transformers with Latent Tokens](https://mlsanity.com/paper/2505.12629/) (2025)
- [Thoughtbubbles: an Unsupervised Method for Parallel Thinking in Latent Space](https://mlsanity.com/paper/2510.00219/) (2025)

## Abstract (arXiv)

Thinking Tokens (TT) have been proposed as an unsupervised method to facilitate reasoning in language models. However, despite their conceptual appeal, our findings show that TTs marginally improves performance and consistently underperforms compared to Chain-of-Thought (CoT) reasoning across multiple benchmarks. We hypothesize that this underperformance stems from the reliance on a single embedding for TTs, which results in inconsistent learning signals and introduces noisy gradients. This paper provides a comprehensive empirical analysis to validate this hypothesis and discusses the implications for future research on unsupervised reasoning in LLMs.