mlsanityData to 7 Oct 2026

Analyzing Encoded Concepts in Transformer Language Models

arXiv 2206.13289 ↗First posted 27 Jun 2022cs.CL cs.AIcited by 37

Not on the map. Why?

Abstract (arXiv)

We propose a novel framework ConceptX, to analyze how latent concepts are encoded in representations learned within pre-trained language models. It uses clustering to discover the encoded concepts and explains them by aligning with a large set of human-defined concepts. Our analysis on seven transformer language models reveal interesting insights: i) the latent space within the learned representations overlap with different linguistic concepts to a varying degree, ii) the lower layers in the model are dominated by lexical concepts (e.g., affixation), whereas the core-linguistic concepts (e.g., morphological or syntactic relations) are better represented in the middle and higher layers, iii) some encoded concepts are multi-faceted and cannot be adequately explained using the existing human-defined concepts.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 40 of its 59 references that are in the map. First posted on arXiv 27 Jun 2022.

Builds on

40 of 59 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  8. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  9. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  10. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  11. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  12. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
28 more
  1. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  2. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  3. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
  4. 2020Concept Whitening for Interpretable Image Recognitioncited by 225
  5. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  6. 2019Visualizing and Measuring the Geometry of BERTcited by 215
  7. 2018Colorless green recurrent networks dream hierarchicallycited by 210
  8. 2015Visualizing and Understanding Neural Models in NLPcited by 199
  9. 2020Compositional Explanations of Neuronscited by 172
  10. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 165
  11. 2015Visualizing and Understanding Recurrent Networkscited by 158
  12. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  13. 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 110
  14. 2020Analyzing Individual Neurons in Pre-trained Language Modelscited by 97
  15. 2018Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Taskscited by 56
  16. 2022Discovering Latent Concepts Learned in BERTcited by 56
  17. 2020Similarity Analysis of Contextual Word Representation Modelscited by 54
  18. 2017The Parallel Meaning Bank: Towards a Multilingual Corpus of Translations Annotated with Compositional Meaning Representationscited by 41
  19. 2016Representation of linguistic form and function in recurrent neural networkscited by 39
  20. 2020Emergence of Separable Manifolds in Deep Language Representationscited by 39
  21. 2018Deep RNNs Encode Soft Hierarchical Syntaxcited by 34
  22. 2019On the Linguistic Representational Power of Neural Machine Translation Modelscited by 34
  23. 2020Asking without Telling: Exploring Latent Ontologies in Contextual Representationscited by 34
  24. 2021How transfer learning impacts linguistic knowledge in deep NLP models?cited by 32
  25. 2018NeuroX: A Toolkit for Analyzing Individual Neurons in Neural Networkscited by 31
  26. 2020Finding Experts in Transformer Modelscited by 28
  27. 2016Word Representation Models for Morphologically Rich Languages in Neural Machine Translationcited by 9
  28. 2021Fine-grained Interpretation and Causation Analysis in Deep NLP Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 7; 2024: 11; 2025: 13; 2026: 4.

37 papers.

Sort:

By area:

  1. 2024Universal Neurons in GPT2 Language Modelscited by 100
  2. 2023LLMs4OL: Large Language Models for Ontology Learningcited by 75
  3. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 46
  4. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  5. 2024An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMscited by 20
  6. 2022On the Transformation of Latent Space in Fine-Tuned NLP Modelscited by 19
  7. 2023Can LLMs facilitate interpretation of pre-trained language models?cited by 15
  8. 2023Towards Explainable Evaluation Metrics for Machine Translationcited by 15
  9. 2024Exploring Alignment in Shared Cross-lingual Spacescited by 13
  10. 2025I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data?cited by 13
  11. 2025Precise In-Parameter Concept Erasure in Large Language Modelscited by 11
  12. 2024Evaluating Readability and Faithfulness of Concept-based Explanationscited by 10
25 more
  1. 2023Scaling up Discovery of Latent Concepts in Deep NLP Modelscited by 9
  2. 2023Learning to Generate Text in Arbitrary Writing Stylescited by 9
  3. 2025ConSim: Measuring Concept-Based Explanations' Effectiveness with Automated Simulatabilitycited by 9
  4. 2025Linguistic Interpretability of Transformer-based Language Models: a systematic reviewcited by 7
  5. 2022ConceptX: A Framework for Latent Concept Analysiscited by 6
  6. 2023Towards Concept-Aware Large Language Modelscited by 6
  7. 2024Tomato, Tomahto, Tomate: Do Multilingual Language Models Understand Based on Subword-Level Semantic Concepts?cited by 4
  8. 2026Tracing Moral Foundations in Large Language Modelscited by 4
  9. 2026Where to Steer: Input-Dependent Layer Selection for Steering Improves LLM Alignmentcited by 3
  10. 2024Adversarial Attacks on Parts of Speech: An Empirical Study in Text-to-Image Generationcited by 2
  11. 2025From Tokens to Lattices: Emergent Lattice Structures in Language Modelscited by 2
  12. 2025The Generalization Ridge: Information Flow in Natural Language Generationcited by 2
  13. 2026The Lattice Representation Hypothesis of Large Language Modelscited by 2
  14. 2023NxPlain: Web-based Tool for Discovery of Latent Conceptscited by 1
  15. 2024Towards a Path Dependent Account of Category Fluencycited by 1
  16. 2025Improve Language Model and Brain Alignment via Associative Memorycited by 1
  17. 2025Enhancing the Comprehensibility of Text Explanations via Unsupervised Concept Discoverycited by 1
  18. 2025From Words to Waves: Analyzing Concept Formation in Speech and Text-Based Foundation Modelscited by 1
  19. 2025Cross-Layer Discrete Concept Discovery for Interpreting Language Modelscited by 1
  20. 2025Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysiscited by 1
  21. 2025On-device Semantic Selection Made Low Latency and Memory Efficient with Monolithic Forwardingcited by 1
  22. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  23. 2024LLM-Oriented Retrieval Tunercited by 0
  24. 2024Investigating OCR-Sensitive Neurons to Improve Entity Recognition in Historical Documentscited by 0
  25. 2025ExpertLens: Activation steering features are highly interpretablecited by 0