# When Models Manipulate Manifolds: The Geometry of a Counting Task

arXiv: https://arxiv.org/abs/2601.04480 · HTML: https://mlsanity.com/paper/2601.04480/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 75 references that are in the map.
- First posted on arXiv 8 Jan 2026.

## Builds on (most cited first, 30 of 47)

- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [Not All Language Model Features Are One-Dimensionally Linear](https://mlsanity.com/paper/2405.14860/) (2024)
- [How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://mlsanity.com/paper/2305.00586/) (2023)
- [Transcoders Find Interpretable LLM Feature Circuits](https://mlsanity.com/paper/2406.11944/) (2024)
- [A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders](https://mlsanity.com/paper/2409.14507/) (2024)
- [The Geometry of Categorical and Hierarchical Concepts in Large Language Models](https://mlsanity.com/paper/2406.01506/) (2024)
- [The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks](https://mlsanity.com/paper/2306.17844/) (2023)
- [The Remarkable Robustness of LLMs: Stages of Inference?](https://mlsanity.com/paper/2406.19384/) (2024)
- [The Quantization Model of Neural Scaling](https://mlsanity.com/paper/2303.13506/) (2023)
- [The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence](https://mlsanity.com/paper/2502.17420/) (2025)
- [Automatically Interpreting Millions of Features in Large Language Models](https://mlsanity.com/paper/2410.13928/) (2024)
- [How do Language Models Bind Entities in Context?](https://mlsanity.com/paper/2310.17191/) (2023)
- [Universal Neurons in GPT2 Language Models](https://mlsanity.com/paper/2401.12181/) (2024)
- [Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics](https://mlsanity.com/paper/2410.21272/) (2024)
- [The Geometry of Multilingual Language Model Representations](https://mlsanity.com/paper/2205.10964/) (2022)
- [Information Flow Routes: Automatically Interpreting Language Models at Scale](https://mlsanity.com/paper/2403.00824/) (2024)
- [A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis](https://mlsanity.com/paper/2305.15054/) (2023)
- [Neurons in Large Language Models: Dead, N-gram, Positional](https://mlsanity.com/paper/2309.04827/) (2023)
- [Language Models Use Trigonometry to Do Addition](https://mlsanity.com/paper/2502.00873/) (2025)
- [A Multimodal Automated Interpretability Agent](https://mlsanity.com/paper/2404.14394/) (2024)
- [Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry](https://mlsanity.com/paper/2503.01822/) (2025)
- [The Geometry of Concepts: Sparse Autoencoder Feature Structure](https://mlsanity.com/paper/2410.19750/) (2024)
- [The Origins of Representation Manifolds in Large Language Models](https://mlsanity.com/paper/2505.18235/) (2025)
- [From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit](https://mlsanity.com/paper/2506.03093/) (2025)
- [LSTM Networks Can Perform Dynamic Counting](https://mlsanity.com/paper/1906.03648/) (2019)

## Built on by (most cited first, 30 of 46)

- [Do Sparse Autoencoders Capture Concept Manifolds?](https://mlsanity.com/paper/2604.28119/) (2026)
- [There Will Be a Scientific Theory of Deep Learning](https://mlsanity.com/paper/2604.21691/) (2026)
- [Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior](https://mlsanity.com/paper/2605.05115/) (2026)
- [Deriving Neural Scaling Laws from the statistics of natural language](https://mlsanity.com/paper/2602.07488/) (2026)
- [The Shape of Beliefs: Geometry, Dynamics, and Interventions along Representation Manifolds of Language Models' Posteriors](https://mlsanity.com/paper/2602.02315/) (2026)
- [From Directions to Regions: Decomposing Activations in Language Models via Local Geometry](https://mlsanity.com/paper/2602.02464/) (2026)
- [The Information Geometry of Softmax: Probing and Steering](https://mlsanity.com/paper/2602.15293/) (2026)
- [Curveball Steering: The Right Direction To Steer Isn't Always Linear](https://mlsanity.com/paper/2603.09313/) (2026)
- [Transformers learn factored representations](https://mlsanity.com/paper/2602.02385/) (2026)
- [The Geometry of Representational Failures in Vision Language Models](https://mlsanity.com/paper/2602.07025/) (2026)
- [Deep networks learn to parse uniform-depth context-free languages from local statistics](https://mlsanity.com/paper/2602.06065/) (2026)
- [Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds](https://mlsanity.com/paper/2606.25234/) (2026)
- [Spectral Superposition: A Theory of Feature Geometry](https://mlsanity.com/paper/2602.02224/) (2026)
- [Decomposing Query-Key Feature Interactions Using Contrastive Covariances](https://mlsanity.com/paper/2602.04752/) (2026)
- [A Geometric Perspective on Next-Token Prediction in Large Language Models: Three Emerging Phases](https://mlsanity.com/paper/2605.09011/) (2026)
- [Finding Belief Geometries with Sparse Autoencoders](https://mlsanity.com/paper/2604.02685/) (2026)
- [Dimensionality-Aware Anomaly Detection in Learned Representations of Self-Supervised Speech Models](https://mlsanity.com/paper/2605.02715/) (2026)
- [Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers](https://mlsanity.com/paper/2605.03780/) (2026)
- [SMIXAE: Towards Unsupervised Manifold Discovery in Language Models](https://mlsanity.com/paper/2605.09224/) (2026)
- [The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling Laws](https://mlsanity.com/paper/2605.09887/) (2026)
- [Tensor Product Representation Probes Reveal Shared Structure Across Linear Directions](https://mlsanity.com/paper/2605.09967/) (2026)
- [Probing for Representation Manifolds in Superposition](https://mlsanity.com/paper/2605.18537/) (2026)
- [Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence](https://mlsanity.com/paper/2605.23821/) (2026)
- [Geometry of Human Perceptual Domains Emerges Transiently in LLM Representations](https://mlsanity.com/paper/2605.27970/) (2026)
- [When and How Long? The Readout-Mediator Angle in Temporal Reasoning](https://mlsanity.com/paper/2605.29126/) (2026)
- [Semantic Delta: An Interpretable Signal Differentiating Human and LLMs Dialogue](https://mlsanity.com/paper/2603.19849/) (2026)
- [Task Ecologies and the Evolution of World-Tracking Representations in Large Language Models](https://mlsanity.com/paper/2604.05469/) (2026)
- [Multilingual Embedding Probes Fail to Generalize Across Learner Corpora](https://mlsanity.com/paper/2604.07095/) (2026)
- [The Position Curse: LLMs Struggle to Locate the Last Few Items in a List](https://mlsanity.com/paper/2605.07127/) (2026)
- [Not How Many, But Which: Parameter Placement in Low-Rank Adaptation](https://mlsanity.com/paper/2605.12207/) (2026)

## Abstract (arXiv)

Language models can perceive visual properties of text despite receiving only sequences of tokens-we mechanistically investigate how Claude 3.5 Haiku accomplishes one such task: linebreaking in fixed-width text. We find that character counts are represented on low-dimensional curved manifolds discretized by sparse feature families, analogous to biological place cells. Accurate predictions emerge from a sequence of geometric transformations: token lengths are accumulated into character count manifolds, attention heads twist these manifolds to estimate distance to the line boundary, and the decision to break the line is enabled by arranging estimates orthogonally to create a linear decision boundary. We validate our findings through causal interventions and discover visual illusions--character sequences that hijack the counting mechanism. Our work demonstrates the rich sensory processing of early layers, the intricacy of attention algorithms, and the importance of combining feature-based and geometric views of interpretability.