# Soft Tokens, Hard Truths

arXiv: https://arxiv.org/abs/2509.19170 · HTML: https://mlsanity.com/paper/2509.19170/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 20 of its 25 references that are in the map.
- First posted on arXiv 23 Sep 2025.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 20 of 20)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach](https://mlsanity.com/paper/2502.05171/) (2025)
- [Think before you speak: Training Language Models With Pause Tokens](https://mlsanity.com/paper/2310.02226/) (2023)
- [Reasoning with Latent Thoughts: On the Power of Looped Transformers](https://mlsanity.com/paper/2502.17416/) (2025)
- [CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation](https://mlsanity.com/paper/2502.21074/) (2025)
- [Let's Think Dot by Dot: Hidden Computation in Transformer Language Models](https://mlsanity.com/paper/2404.15758/) (2024)
- [Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space](https://mlsanity.com/paper/2505.15778/) (2025)
- [Outcome-based Exploration for LLM Reasoning](https://mlsanity.com/paper/2509.06941/) (2025)
- [Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought](https://mlsanity.com/paper/2505.12514/) (2025)
- [LLMs are Single-threaded Reasoners: Demystifying the Working Mechanism of Soft Thinking](https://mlsanity.com/paper/2508.03440/) (2025)
- [Learning a Continue-Thinking Token for Enhanced Test-Time Scaling](https://mlsanity.com/paper/2506.11274/) (2025)

## Built on by (most cited first, 30 of 35)

- [Monet: Reasoning in Latent Visual Space Beyond Images and Language](https://mlsanity.com/paper/2511.21395/) (2025)
- [LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning](https://mlsanity.com/paper/2510.04573/) (2025)
- [Reasoning Within the Mind: Dynamic Multimodal Interleaving in Latent Space](https://mlsanity.com/paper/2512.12623/) (2025)
- [SofT-GRPO: Surpassing Discrete-Token LLM Reinforcement Learning via Gumbel-Reparameterized Soft-Thinking Policy Optimization](https://mlsanity.com/paper/2511.06411/) (2025)
- [Generative Recursive Reasoning](https://mlsanity.com/paper/2605.19376/) (2026)
- [Multiplex Thinking: Reasoning via Token-wise Branch-and-Merge](https://mlsanity.com/paper/2601.08808/) (2026)
- [Latent-GRPO: Group Relative Policy Optimization for Latent Reasoning](https://mlsanity.com/paper/2604.27998/) (2026)
- [ThinkRouter: Efficient Reasoning via Routing Thinking between Latent and Discrete Spaces](https://mlsanity.com/paper/2602.11683/) (2026)
- [Thinking Without Words: Efficient Latent Reasoning with Abstract Chain-of-Thought](https://mlsanity.com/paper/2604.22709/) (2026)
- [LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learning](https://mlsanity.com/paper/2602.01705/) (2026)
- [The Illusion of Superposition? A Principled Analysis of Latent Thinking in Language Models](https://mlsanity.com/paper/2604.06374/) (2026)
- [State over Tokens: Characterizing the Role of Reasoning Tokens](https://mlsanity.com/paper/2512.12777/) (2025)
- [Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism](https://mlsanity.com/paper/2510.13170/) (2025)
- [Latent Thoughts Tuning: Bridging Context and Reasoning with Fused Information in Latent Tokens](https://mlsanity.com/paper/2602.10229/) (2026)
- [LEPO: Latent Reasoning Policy Optimization for Large Language Models](https://mlsanity.com/paper/2604.17892/) (2026)
- [Latent Reasoning with Normalizing Flows](https://mlsanity.com/paper/2606.06447/) (2026)
- [DiscoLoop: Looping Discrete Embeddings and Continuous Hidden States for Multi-hop Reasoning](https://mlsanity.com/paper/2607.00341/) (2026)
- [Learning to Reason with Mixture of Tokens](https://mlsanity.com/paper/2509.21482/) (2025)
- [Self-Consistent Latent Reasoning: Long Latent Sequence Reasoning for Vision-Language Model](https://mlsanity.com/paper/2605.12163/) (2026)
- [What Makes Effective Supervision in Latent Chain-of-Thought: An Information-Theoretic Analysis](https://mlsanity.com/paper/2606.20075/) (2026)
- [BFS-PO: Best-First Search for Large Reasoning Models](https://mlsanity.com/paper/2602.14917/) (2026)
- [Efficient and Trainable Language Model Test-Time Scaling via Local Branch Routing](https://mlsanity.com/paper/2606.25354/) (2026)
- [Training Continuous Chain of Thought Models: A Tale of Two Regimes](https://mlsanity.com/paper/2607.16972/) (2026)
- [Multimodal Continuous Reasoning via Asymmetric Mutual Variational Learning](https://mlsanity.com/paper/2607.00461/) (2026)
- [GeoAnchor: Collaborative Reasoning via Latent Decomposition for 3D Spatial Understanding](https://mlsanity.com/paper/2607.13454/) (2026)
- [TARPO: Token-Wise Latent-Explicit Reasoning via Action-Routing Policy Optimization](https://mlsanity.com/paper/2606.05859/) (2026)
- [One Token per Multimodal Evidence: Latent Memory for Resource-Constrained QA](https://mlsanity.com/paper/2606.10572/) (2026)
- [N-GRPO: Embedding-Level Neighbor Mixing for Enhanced Policy Optimization](https://mlsanity.com/paper/2606.10768/) (2026)
- [Soft Token Alignment for Cross-Lingual Reasoning](https://mlsanity.com/paper/2606.26466/) (2026)
- [A Model with No Head and Many Thoughts](https://mlsanity.com/paper/2608.31069/) (2026)

## Abstract (arXiv)

The use of continuous instead of discrete tokens during the Chain-of-Thought (CoT) phase of reasoning LLMs has garnered attention recently, based on the intuition that a continuous mixture of discrete tokens could simulate a superposition of several reasoning paths simultaneously. Theoretical results have formally proven that continuous tokens have much greater expressivity and can solve specific problems more efficiently. However, practical use of continuous tokens has been limited by strong training difficulties: previous works either just use continuous tokens at inference time on a pre-trained discrete-token model, or must distill the continuous CoT from ground-truth discrete CoTs and face computational costs that limit the CoT to very few tokens. This is the first work introducing a scalable method to learn continuous CoTs via reinforcement learning (RL), without distilling from reference discrete CoTs. We use "soft" tokens: mixtures of tokens together with noise on the input embedding to provide RL exploration. Computational overhead is minimal, enabling us to learn continuous CoTs with hundreds of tokens. On math reasoning benchmarks with Llama and Qwen models up to 8B, training with continuous CoTs match discrete-token CoTs for pass@1 and surpass them for pass@32, showing greater CoT diversity. In systematic comparisons, the best-performing scenario is to train with continuous CoT tokens then use discrete tokens for inference, meaning the "soft" models can be deployed in a standard way. Finally, we show continuous CoT RL training better preserves the predictions of the base model on out-of-domain tasks, thus providing a softer touch to the base model.