mlsanityData to 14 Oct 2026

Latent Concept Disentanglement in Transformer-based Language Models

arXiv 2506.16975 ↗First posted 20 Jun 2025cs.LG cs.AI cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

When large language models (LLMs) use in-context learning (ICL) to solve a new task, they must infer latent concepts from demonstration examples. This raises the question of whether and how transformers represent latent structures as part of their computation. Our work experiments with several controlled tasks, studying this question using mechanistic interpretability. First, we show that in transitive reasoning tasks with a latent, discrete concept, the model successfully identifies the latent concept and does step-by-step concept composition. This builds upon prior work that analyzes single-step reasoning. Then, we consider tasks parameterized by a latent numerical concept. We discover low-dimensional subspaces in the model's representation space, where the geometry cleanly reflects the underlying parameterization. Overall, we show that small and large models can indeed disentangle and utilize latent concepts that they learn in-context from a handful of abbreviated demonstrations.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 49 of its 66 references that are in the map. First posted on arXiv 20 Jun 2025.

Builds on

49 of 66 references.

References · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  3. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  4. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  5. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  6. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  7. 2022Editing Models with Task Arithmeticcited by 1,120
  8. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  9. 2022In-context Learning and Induction Headscited by 964
  10. 2022Transformers learn in-context by gradient descentcited by 732
  11. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  12. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
37 more
  1. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  2. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  3. 2025Reasoning Models Don't Always Say What They Thinkcited by 415
  4. 2024The Platonic Representation Hypothesiscited by 382
  5. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  6. 2023In-Context Learning Creates Task Vectorscited by 312
  7. 2023Function Vectors in Large Language Modelscited by 308
  8. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  9. 2013Direct and Indirect Effectscited by 280
  10. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  11. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 169
  12. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 159
  13. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 118
  14. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 103
  15. 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 98
  16. 2025Model Organisms for Emergent Misalignmentcited by 91
  17. 2025Convergent Linear Representations of Emergent Misalignmentcited by 71
  18. 2025Language Models Use Trigonometry to Do Additioncited by 70
  19. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 63
  20. 2025Which Attention Heads Matter for In-Context Learning?cited by 61
  21. 2024Competition Dynamics Shape Algorithmic Phases of In-Context Learningcited by 54
  22. 2024The Geometry of Concepts: Sparse Autoencoder Feature Structurecited by 52
  23. 2024A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Taskcited by 49
  24. 2024Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformerscited by 44
  25. 2024Dual Operating Modes of In-Context Learningcited by 43
  26. 2024Pre-trained Large Language Models Use Fourier Features to Compute Additioncited by 39
  27. 2024The LLM Language Network: A Neuroscientific Approach for Identifying Causally Task-Relevant Unitscited by 34
  28. 2024Transformers on Markov Data: Constant Depth Sufficescited by 27
  29. 2025Towards Understanding Distilled Reasoning Models: A Representational Approachcited by 26
  30. 2025How Do LLMs Perform Two-Hop Reasoning in Context?cited by 20
  31. 2025Strategy Coopetition Explains the Emergence and Transience of In-Context Learningcited by 17
  32. 2025Do different prompting methods yield a common task representation in language models?cited by 11
  33. 2025Contextualize-then-Aggregate: Circuits for In-Context Learning in Gemma-2 2Bcited by 10
  34. 2025Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergencecited by 10
  35. 2025Do Sparse Autoencoders Generalize? A Case Study of Answerabilitycited by 8
  36. 2025Understanding In-context Learning of Addition via Activation Subspacescited by 8
  37. 2025Functional Abstraction of Knowledge Recall in Large Language Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Learning without training: The implicit dynamics of in-context learningcited by 43
  2. 2026In-Context Learning Operates as Concept Subspace Learningcited by 3
  3. 2025The Knowledge-Reasoning Dissociation: Fundamental Limitations of LLMs in Clinical Natural Language Inferencecited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Latent Reasoning
  2. 2026Large Reasoning Models Are (Not Yet) Multilingual Latent Reasoners
  3. 2026Unlocking the Black Box of Latent Reasoning: An Interpretability-Guided Approach to Intervention
  4. 2026LLM Reasoning Is Latent, Not the Chain of Thought
  5. 2025Learning to Disentangle Latent Reasoning Rules with Language VAEs: A Systematic Study
  6. 2026Why Struggle with Continuous Latents? Interpretable Discrete Latent Reasoning via Rendered Compression
  7. 2025From Words to Waves: Analyzing Concept Formation in Speech and Text-Based Foundation Models
  8. 2026MechELK: A Mechanistic Interpretability Framework for Eliciting Latent Knowledge in Large Language Models