mlsanityData to 7 Oct 2026

The Geometry of Categorical and Hierarchical Concepts in Large Language Models

arXiv 2406.01506 ↗First posted 3 Jun 2024cs.CL cs.AI cs.LGcited by 151

Not on the map. Why?

Abstract (arXiv)

The linear representation hypothesis is the informal idea that semantic concepts are encoded as linear directions in the representation spaces of large language models (LLMs). Previous work has shown how to make this notion precise for representing binary concepts that have natural contrasts (e.g., {male, female}) as directions in representation space. However, many natural concepts do not have natural contrasts (e.g., whether the output is about an animal). In this work, we show how to extend the formalization of the linear representation hypothesis to represent features (e.g., is_animal) as vectors. This allows us to immediately formalize the representation of categorical concepts as polytopes in the representation space. Further, we use the formalization to prove a relationship between the hierarchical structure of concepts and the geometry of their representations. We validate these theoretical results on the Gemma and LLaMA-3 large language models, estimating representations for 900+ hierarchically related concepts using data from WordNet.

Key facts. Cited by 151 papers in the map (data through 7 Oct 2026). Builds on 38 of its 56 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

38 of 56 references.

References · 38

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  3. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  4. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  5. 2022Toy Models of Superpositioncited by 928
  6. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  7. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  8. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  9. 2017Poincar'e Embeddings for Learning Hierarchical Representationscited by 674
  10. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  11. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  12. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
26 more
  1. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  2. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  3. 2023Language Models Represent Space and Timecited by 306
  4. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  5. 2019Visualizing and Measuring the Geometry of BERTcited by 215
  6. 2022Relative representations enable zero-shot latent space communicationcited by 175
  7. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  8. 2018Hyperbolic Entailment Cones for Learning Hierarchical Embeddingscited by 160
  9. 2023Linear Representations of Sentiment in Large Language Modelscited by 157
  10. 2023The geometry of hidden representations of large transformer modelscited by 108
  11. 2022The Geometry of Multilingual Language Model Representationscited by 94
  12. 2024On the Origins of Linear Representations in Large Language Modelscited by 76
  13. 2024Identifying Functionally Important Features with End-to-End Sparse Dictionary Learningcited by 64
  14. 2019Analogies Explained: Towards Understanding Word Embeddingscited by 63
  15. 2023Concept Algebra for (Score-Based) Text-Controlled Generative Modelscited by 59
  16. 2018Towards Understanding Linear Word Analogiescited by 57
  17. 2021Probing BERT in Hyperbolic Spacescited by 50
  18. 2024Language Models as Hierarchy Encoderscited by 26
  19. 2023Uncovering Meanings of Embeddings via Partial Orthogonalitycited by 15
  20. 2016Exponential Family Embeddingscited by 13
  21. 2018What the Vec? Towards Probabilistically Grounded Embeddingscited by 8
  22. 2017Dynamic Bernoulli Embeddings for Language Evolutioncited by 6
  23. 2019Understanding Composition of Word Embeddings via Tensor Decompositioncited by 4
  24. 2019Natural Alpha Embeddingscited by 4
  25. 2022Learning general conditional independence structures via the neighbourhood latticecited by 4
  26. 2023Contrastive Loss is All You Need to Recover Analogies as Parallel Linescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 44; 2026: 94.

151 papers.

Sort:

By area:

  1. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  2. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometrycited by 58
  3. 2024The Geometry of Concepts: Sparse Autoencoder Feature Structurecited by 52
  4. 2024ICLR: In-Context Learning of Representationscited by 49
  5. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 46
  6. 2025The Origins of Representation Manifolds in Large Language Modelscited by 46
  7. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuitcited by 45
  8. 2026When Models Manipulate Manifolds: The Geometry of a Counting Taskcited by 43
  9. 2025Shared Global and Local Geometry of Language Model Embeddingscited by 34
  10. 2025Towards Understanding Distilled Reasoning Models: A Representational Approachcited by 26
  11. 2024Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoderscited by 25
  12. 2026Do Sparse Autoencoders Capture Concept Manifolds?cited by 25
38 more
  1. 2024Beyond Single Concept Vector: Modeling Concept Subspace in LLMs with Gaussian Distributioncited by 24
  2. 2025Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steeringcited by 24
  3. 2025From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaningcited by 23
  4. 2026Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behaviorcited by 22
  5. 2025The Geometry of Reasoning: Flowing Logics in Representation Spacecited by 21
  6. 2025From superposition to sparse codes: interpretable representations in neural networkscited by 16
  7. 2025Differentially Private Steering for Large Language Model Alignmentcited by 15
  8. 2025Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometriescited by 15
  9. 2024The representation landscape of few-shot learning and fine-tuning in large language modelscited by 14
  10. 2024Geometric Signatures of Compositionality Across a Language Model's Lifetimecited by 14
  11. 2024All or None: Identifiable Linear Properties of Next-token Predictors in Language Modelingcited by 14
  12. 2026Observing and Controlling Features in Vision-Language-Action Modelscited by 14
  13. 2025I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data?cited by 13
  14. 2025Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failurecited by 13
  15. 2025Learning Safety Constraints for Large Language Modelscited by 13
  16. 2025Incorporating Hierarchical Semantics in Sparse Autoencoder Architecturescited by 12
  17. 2025Video Unlearning via Low-Rank Refusal Vectorcited by 11
  18. 2026From Directions to Regions: Decomposing Activations in Language Models via Local Geometrycited by 10
  19. 2025On the Emergence of Linear Analogies in Word Embeddingscited by 9
  20. 2025The Blessing and Curse of Dimensionality in Safety Alignmentcited by 9
  21. 2025Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editingcited by 9
  22. 2026Mechanistic Evidence for Faithfulness Decay in Chain-of-Thought Reasoningcited by 9
  23. 2026The Information Geometry of Softmax: Probing and Steeringcited by 9
  24. 2026From Data Statistics to Feature Geometry: How Correlations Shape Superpositioncited by 9
  25. 2025From Papers to Panoramas: Building Hierarchies of Scientific Literature at Scalecited by 7
  26. 2025The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognitioncited by 7
  27. 2025Semantic Structure in Large Language Model Embeddingscited by 7
  28. 2025OntoURL: A Benchmark for Evaluating Large Language Models on Symbolic Ontological Understanding, Reasoning and Learningcited by 6
  29. 2025On the Predictive Power of Representation Dispersion in Language Modelscited by 6
  30. 2025Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter Itcited by 6
  31. 2025Hypothesis-Driven Feature Manifold Analysis in LLMs via Supervised Multi-Dimensional Scalingcited by 6
  32. 2025Semantic Surgery: Zero-Shot Concept Erasure in Diffusion Modelscited by 6
  33. 2026Differential syntactic and semantic encoding in LLMscited by 6
  34. 2026From Kepler to Newton: Inductive Biases Guide Learned World Models in Transformerscited by 6
  35. 2026Semantic Tube Prediction: Beating LLM Data Efficiency with JEPAcited by 6
  36. 2024Frame Representation Hypothesis: Multi-Token LLM Interpretability and Concept-Guided Text Generationcited by 5
  37. 2025Bayesian Fields: Task-driven Open-Set Semantic Gaussian Splattingcited by 5
  38. 2025Specialization after Generalization: Towards Understanding Test-Time Training in Foundation Modelscited by 5

50 most cited of 151 All 151 (JSON)