# Planting a Latent Variable in Natural-Looking Text: a More Realistic Test of Belief States in LLMs and Their Link to Concept Geometry

arXiv: https://arxiv.org/abs/2608.26887 · HTML: https://mlsanity.com/paper/2608.26887/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 8 references that are in the map.
- First posted on arXiv 27 Aug 2026.

## Builds on (most cited first, 5 of 5)

- [Not All Language Model Features Are One-Dimensionally Linear](https://mlsanity.com/paper/2405.14860/) (2024)
- [Transformers represent belief state geometry in their residual stream](https://mlsanity.com/paper/2405.15943/) (2024)
- [Symmetry in language statistics shapes the geometry of model representations](https://mlsanity.com/paper/2602.15029/) (2026)
- [The Shape of Beliefs: Geometry, Dynamics, and Interventions along Representation Manifolds of Language Models' Posteriors](https://mlsanity.com/paper/2602.02315/) (2026)
- [Stories in Space: In-Context Learning Trajectories in Conceptual Belief Space](https://mlsanity.com/paper/2605.12412/) (2026)

## Abstract (arXiv)

LLMs are thought to track "belief states," i.e., running probability distributions over the latent variables that govern language (Shai et al., 2024; Sarfati et al., 2026), but so far this has only been comprehensively demonstrated on toy synthetic data and in a few isolated case studies. It has also never been empirically connected to the geometry of LLM features (the concepts interpretability finds in model activations). In this work, we plant a controllable latent variable inside natural-looking text. An LLM teacher writes ordinary text while we "subliminally" steer it along one of K = 8 unrelated sparse autoencoder directions at each token, with the active directions following a ring-shaped Markov chain. A small transformer model trained on this corpus does indeed track the Bayesian posterior belief about our planted latent variable. Moreover, it also arranges the 8 states themselves on a ring, in the exact order of the Markov chain, which is supporting evidence that a concept's geometry can be formed by the statistical dynamics of the latent variable behind it.