# Latent Thought Models with Variational Bayes Inference-Time Computation

arXiv: https://arxiv.org/abs/2502.01567 · HTML: https://mlsanity.com/paper/2502.01567/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 47 of its 73 references that are in the map.
- First posted on arXiv 3 Feb 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [Simple and Effective Masked Diffusion Language Models](https://mlsanity.com/paper/2406.07524/) (2024)
- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution](https://mlsanity.com/paper/2310.16834/) (2023)
- [Simplified and Generalized Masked Diffusion for Discrete Data](https://mlsanity.com/paper/2406.04329/) (2024)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling](https://mlsanity.com/paper/1312.3005/) (2013)
- [Beam Search Strategies for Neural Machine Translation](https://mlsanity.com/paper/1702.01806/) (2017)
- [Continuous diffusion for categorical data](https://mlsanity.com/paper/2211.15089/) (2022)
- [Autoregressive Diffusion Models](https://mlsanity.com/paper/2110.02037/) (2021)
- [Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling](https://mlsanity.com/paper/2409.02908/) (2024)
- [Bayesian Model-Agnostic Meta-Learning](https://mlsanity.com/paper/1806.03836/) (2018)
- [SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control](https://mlsanity.com/paper/2210.17432/) (2022)

## Built on by (most cited first, 25 of 25)

- [Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models](https://mlsanity.com/paper/2503.09573/) (2025)
- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?](https://mlsanity.com/paper/2503.24235/) (2025)
- [Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation](https://mlsanity.com/paper/2507.10524/) (2025)
- [Reasoning to Learn from Latent Thoughts](https://mlsanity.com/paper/2503.18866/) (2025)
- [Implicit Reasoning in Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2509.02350/) (2025)
- [A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Law](https://mlsanity.com/paper/2505.02665/) (2025)
- [NeuReasoner: Towards Explainable, Controllable, and Unified Reasoning via Mixture-of-Neurons](https://mlsanity.com/paper/2604.02972/) (2026)
- [Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospects](https://mlsanity.com/paper/2505.20223/) (2025)
- [Revisiting Test-Time Scaling: A Survey and a Diversity-Aware Method for Efficient Reasoning](https://mlsanity.com/paper/2506.04611/) (2025)
- [T^2MLR: Transformer with Temporal Middle-Layer Recurrence](https://mlsanity.com/paper/2607.15178/) (2026)
- [Latent Adaptive Planner for Dynamic Manipulation](https://mlsanity.com/paper/2505.03077/) (2025)
- [Capabilities and Fundamental Limits of Latent Chain-of-Thought](https://mlsanity.com/paper/2602.01148/) (2026)
- [Latent Thought Flow: Efficient Latent Reasoning in Large Language Models](https://mlsanity.com/paper/2606.16222/) (2026)
- [Generative Actor Critic](https://mlsanity.com/paper/2512.21527/) (2025)
- [Modeling Language as a Sequence of Thoughts](https://mlsanity.com/paper/2512.25026/) (2025)
- [BFS-PO: Best-First Search for Large Reasoning Models](https://mlsanity.com/paper/2602.14917/) (2026)
- [Learned Relay Representations for Forward-Thinking Discrete Diffusion Models](https://mlsanity.com/paper/2605.22967/) (2026)
- [Thoughts-as-Planning: Latent World Models for Chain-of-Thoughts Optimization via Reinforcement Planning](https://mlsanity.com/paper/2605.28842/) (2026)
- [Reasoning as Energy Minimization over Structured Latent Trajectories](https://mlsanity.com/paper/2603.28248/) (2026)
- [Forecasting Is Not Attribution: Localizing Decoder Bypass in Graph-Based Neural Marketing Mix Models](https://mlsanity.com/paper/2606.12687/) (2026)
- [Uncovering Latent Reasoning Strategies in Language Models](https://mlsanity.com/paper/2607.17674/) (2026)
- [Representation-based Masked Diffusion Model](https://mlsanity.com/paper/2609.12382/) (2026)
- [Next Thoughts Are Distributions: Generative Autoregressive Reasoning in the Latent Space](https://mlsanity.com/paper/2609.33271/) (2026)
- [Learning to Plan from Random Exploration](https://mlsanity.com/paper/2609.38383/) (2026)

## Abstract (arXiv)

We propose a novel class of language models, Latent Thought Models (LTMs), which incorporate explicit latent thought vectors that follow an explicit prior model in latent space. These latent thought vectors guide the autoregressive generation of ground tokens through a Transformer decoder. Training employs a dual-rate optimization process within the classical variational Bayes framework: fast learning of local variational parameters for the posterior distribution of latent vectors (inference-time computation), and slow learning of global decoder parameters. Empirical studies reveal that LTMs possess additional scaling dimensions beyond traditional Large Language Models (LLMs), such as the number of iterations in inference-time computation and number of latent thought vectors. Higher sample efficiency can be achieved by increasing training compute per token, with further gains possible by trading model size for more inference steps. Designed based on these scaling properties, LTMs demonstrate superior sample and parameter efficiency compared to autoregressive models and discrete diffusion models. They significantly outperform these counterparts in validation perplexity and zero-shot language modeling tasks. Additionally, LTMs exhibit emergent few-shot in-context reasoning capabilities that scale with model size, and achieve competitive performance in conditional and unconditional text generation.