mlsanityData to 7 Oct 2026

When Models Manipulate Manifolds: The Geometry of a Counting Task

arXiv 2601.04480 ↗First posted 8 Jan 2026cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

Language models can perceive visual properties of text despite receiving only sequences of tokens-we mechanistically investigate how Claude 3.5 Haiku accomplishes one such task: linebreaking in fixed-width text. We find that character counts are represented on low-dimensional curved manifolds discretized by sparse feature families, analogous to biological place cells. Accurate predictions emerge from a sequence of geometric transformations: token lengths are accumulated into character count manifolds, attention heads twist these manifolds to estimate distance to the line boundary, and the decision to break the line is enabled by arranging estimates orthogonally to create a linear decision boundary. We validate our findings through causal interventions and discover visual illusions--character sequences that hijack the counting mechanism. Our work demonstrates the rich sensory processing of early layers, the intricacy of attention algorithms, and the importance of combining feature-based and geometric views of interpretability.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 47 of its 75 references that are in the map. First posted on arXiv 8 Jan 2026.

Builds on

47 of 75 references.

References · 47

  1. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  2. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  3. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  4. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  5. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  6. 2019Visualizing and Measuring the Geometry of BERTcited by 215
  7. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 201
  8. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  9. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 186
  10. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  11. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 151
  12. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoderscited by 148
35 more
  1. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  2. 2023The Quantization Model of Neural Scalingcited by 137
  3. 2024Automatically Interpreting Millions of Features in Large Language Modelscited by 106
  4. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 104
  5. 2023How do Language Models Bind Entities in Context?cited by 102
  6. 2024Universal Neurons in GPT2 Language Modelscited by 100
  7. 2022The Geometry of Multilingual Language Model Representationscited by 94
  8. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 94
  9. 2024Information Flow Routes: Automatically Interpreting Language Models at Scalecited by 92
  10. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 88
  11. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  12. 2025Language Models Use Trigonometry to Do Additioncited by 68
  13. 2024A Multimodal Automated Interpretability Agentcited by 62
  14. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometrycited by 58
  15. 2024The Geometry of Concepts: Sparse Autoencoder Feature Structurecited by 52
  16. 2025The Origins of Representation Manifolds in Large Language Modelscited by 46
  17. 2019LSTM Networks Can Perform Dynamic Countingcited by 45
  18. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuitcited by 45
  19. 2025Enhancing Automated Interpretability with Output-Centric Feature Descriptionscited by 39
  20. 2024Pre-trained Large Language Models Use Fourier Features to Compute Additioncited by 38
  21. 2024The Missing Curve Detectors of InceptionV1: Applying Sparse Autoencoders to InceptionV1 Early Visioncited by 37
  22. 2023Feature emergence via margin maximization: case studies in algebraic taskscited by 33
  23. 2025(How) Do Language Models Track State?cited by 31
  24. 2024Language Models Need Inductive Biases to Count Inductivelycited by 25
  25. 2024Relational Composition in Neural Networks: A Survey and Call to Actioncited by 23
  26. 2024Automatically Identifying Local and Global Circuits with Linear Computation Graphscited by 22
  27. 2024Monotonic Representation of Numeric Properties in Language Modelscited by 15
  28. 2024Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objectscited by 12
  29. 2024Position: An Inner Interpretability Framework for AI Inspired by Lessons from Cognitive Neurosciencecited by 11
  30. 2025Understanding sparse autoencoder scaling in the presence of feature manifoldscited by 11
  31. 2025Understanding In-context Learning of Addition via Activation Subspacescited by 8
  32. 2025Decomposing Representation Space into Interpretable Subspaces with Unsupervised Learningcited by 7
  33. 2024Multilevel Interpretability Of Artificial Neural Networks: Leveraging Framework And Methods From Neurosciencecited by 5
  34. 2025Number Representations in LLMs: A Computational Parallel to Human Perceptioncited by 4
  35. 2025Length Representations in Large Language Modelscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 43.

43 papers.

Sort:

By area:

  1. 2026Do Sparse Autoencoders Capture Concept Manifolds?cited by 25
  2. 2026Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behaviorcited by 22
  3. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  4. 2026Deriving Neural Scaling Laws from the statistics of natural languagecited by 18
  5. 2026The Shape of Beliefs: Geometry, Dynamics, and Interventions along Representation Manifolds of Language Models' Posteriorscited by 15
  6. 2026From Directions to Regions: Decomposing Activations in Language Models via Local Geometrycited by 10
  7. 2026The Information Geometry of Softmax: Probing and Steeringcited by 9
  8. 2026The Geometry of Representational Failures in Vision Language Modelscited by 5
  9. 2026Curveball Steering: The Right Direction To Steer Isn't Always Linearcited by 5
  10. 2026Deep networks learn to parse uniform-depth context-free languages from local statisticscited by 4
  11. 2026Spectral Superposition: A Theory of Feature Geometrycited by 3
  12. 2026Transformers learn factored representationscited by 3
31 more
  1. 2026Decomposing Query-Key Feature Interactions Using Contrastive Covariancescited by 3
  2. 2026A Geometric Perspective on Next-Token Prediction in Large Language Models: Three Emerging Phasescited by 3
  3. 2026Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifoldscited by 3
  4. 2026Finding Belief Geometries with Sparse Autoencoderscited by 2
  5. 2026Dimensionality-Aware Anomaly Detection in Learned Representations of Self-Supervised Speech Modelscited by 2
  6. 2026SMIXAE: Towards Unsupervised Manifold Discovery in Language Modelscited by 1
  7. 2026Geometry of Human Perceptual Domains Emerges Transiently in LLM Representationscited by 1
  8. 2026When and How Long? The Readout-Mediator Angle in Temporal Reasoningcited by 1
  9. 2026Semantic Delta: An Interpretable Signal Differentiating Human and LLMs Dialoguecited by 0
  10. 2026Task Ecologies and the Evolution of World-Tracking Representations in Large Language Modelscited by 0
  11. 2026Multilingual Embedding Probes Fail to Generalize Across Learner Corporacited by 0
  12. 2026The Position Curse: LLMs Struggle to Locate the Last Few Items in a Listcited by 0
  13. 2026The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling Lawscited by 0
  14. 2026Tensor Product Representation Probes Reveal Shared Structure Across Linear Directionscited by 0
  15. 2026Not How Many, But Which: Parameter Placement in Low-Rank Adaptationcited by 0
  16. 2026Non-linear Interventions on Large Language Modelscited by 0
  17. 2026Probing for Representation Manifolds in Superpositioncited by 0
  18. 2026Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrencecited by 0
  19. 2026Structure and Scale in Simplicial Sequence Modellingcited by 0
  20. 2026Adversarial Concept Search: Predicting Compositional Errors From Feature Geometrycited by 0
  21. 2026Geometry of Ordinal Representations in Language Modelscited by 0
  22. 2026Laguerre Geometry for Interpreting Large Language Modelscited by 0
  23. 2026A Shared Subcircuit Lets LLMs Count Down Across Taskscited by 0
  24. 2026Sky sphere representation in language modelscited by 0
  25. 2026What, Where, and How: Probing Spatiotemporal Representations in Video Foundation Modelscited by 0
  26. 2026MAxBench: A Multinomial Concept Recovery Benchmarkcited by 0
  27. 2026LAYERSCOPE: A Layerwise Characterization of Video and Multimodal Learned Representationscited by 0
  28. 2026Late Attention Layers Alone Can Copy Entity Tokens, but Not Without Attending to Their Contextcited by 0
  29. 2026Causal and Interpretable Structures in LLM Compositional Taskscited by 0
  30. 2026When Models Don't Manipulate Manifolds: The Geometry of a Comparison Taskcited by 0
  31. 2026Geometric Similarity in VLM Low-Level Vision Representationscited by 0