mlsanityData to 7 Oct 2026

Scaling up Discovery of Latent Concepts in Deep NLP Models

arXiv 2308.10263 ↗First posted 20 Aug 2023cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Despite the revolution caused by deep NLP models, they remain black boxes, necessitating research to understand their decision-making processes. A recent work by Dalvi et al. (2022) carried out representation analysis through the lens of clustering latent spaces within pre-trained models (PLMs), but that approach is limited to small scale due to the high cost of running Agglomerative hierarchical clustering. This paper studies clustering algorithms in order to scale the discovery of encoded concepts in PLM representations to larger datasets and models. We propose metrics for assessing the quality of discovered latent concepts and use them to compare the studied clustering algorithms. We found that K-Means-based concept discovery significantly enhances efficiency while maintaining the quality of the obtained concepts. Furthermore, we demonstrate the practicality of this newfound efficiency by scaling latent concept discovery to LLMs and phrasal concepts.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 29 of its 36 references that are in the map. First posted on arXiv 20 Aug 2023.

Builds on

29 of 36 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  7. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  8. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  9. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
  10. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  11. 2019Revealing the Dark Secrets of BERTcited by 300
  12. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 165
17 more
  1. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  2. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  3. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  4. 2022Discovering Latent Concepts Learned in BERTcited by 56
  5. 2017The Parallel Meaning Bank: Towards a Multilingual Corpus of Translations Annotated with Compositional Meaning Representationscited by 41
  6. 2020Topic Modeling with Contextualized Word Representation Clusterscited by 41
  7. 2022Analyzing Encoded Concepts in Transformer Language Modelscited by 37
  8. 2020Asking without Telling: Exploring Latent Ontologies in Contextual Representationscited by 34
  9. 2021How transfer learning impacts linguistic knowledge in deep NLP models?cited by 32
  10. 2022Discovering Salient Neurons in Deep NLP Modelscited by 21
  11. 2022On the Transformation of Latent Space in Fine-Tuned NLP Modelscited by 19
  12. 2023Can LLMs facilitate interpretation of pre-trained language models?cited by 15
  13. 2022Beyond Prompting: Making Pre-trained Language Models Better Zero-shot Learners by Clustering Representationscited by 11
  14. 2023NeuroX Library for Neuron Analysis of Deep NLP Modelscited by 11
  15. 2020Deep Clustering of Text Representations for Supervision-free Probing of Syntaxcited by 7
  16. 2022Latent Topology Induction for Understanding Contextualized Representationscited by 7
  17. 2022ConceptX: A Framework for Latent Concept Analysiscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 3.

9 papers.

  1. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  2. 2024Latent Concept-based Explanation of NLP Modelscited by 13
  3. 2024Exploring Alignment in Shared Cross-lingual Spacescited by 13
  4. 2025From Tokens to Lattices: Emergent Lattice Structures in Language Modelscited by 2
  5. 2026The Lattice Representation Hypothesis of Large Language Modelscited by 2
  6. 2025From Words to Waves: Analyzing Concept Formation in Speech and Text-Based Foundation Modelscited by 1
  7. 2025Analyzing Latent Concepts in Code Language Modelscited by 1
  8. 2026Vector Quantized Latent Concepts: A Scalable Alternative to Clustering-Based Concept Discoverycited by 0
  9. 2026Latent Cluster Analysis for Vision-Language-Action Modelscited by 0