mlsanityData to 14 Oct 2026

Can LLMs facilitate interpretation of pre-trained language models?

arXiv 2305.13386 ↗First posted 22 May 2023cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Work done to uncover the knowledge encoded within pre-trained language models rely on annotated corpora or human-in-the-loop methods. However, these approaches are limited in terms of scalability and the scope of interpretation. We propose using a large language model, ChatGPT, as an annotator to enable fine-grained interpretation analysis of pre-trained language models. We discover latent concepts within pre-trained language models by applying agglomerative hierarchical clustering over contextualized representations and then annotate these concepts using ChatGPT. Our findings demonstrate that ChatGPT produces accurate and semantically richer annotations compared to human-annotated concepts. Additionally, we showcase how GPT-based annotations empower interpretation analysis methodologies of which we demonstrate two: probing frameworks and neuron interpretation. To facilitate further exploration and experimentation in the field, we make available a substantial ConceptNet dataset (TCN) comprising 39,000 annotated concepts.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 45 of its 56 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

45 of 56 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  11. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
33 more
  1. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  2. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  3. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  4. 2019Designing and Interpreting Probes with Control Taskscited by 640
  5. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  6. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 481
  7. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  8. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
  9. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  10. 2022Is GPT-3 a Good Data Annotator?cited by 208
  11. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  12. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 166
  13. 2015Visualizing and Understanding Recurrent Networkscited by 160
  14. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  15. 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 111
  16. 2022ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reportscited by 101
  17. 2019The emergence of number and syntax units in LSTM language modelscited by 94
  18. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  19. 2022How would Stance Detection Techniques Evolve after the Launch of ChatGPT?cited by 76
  20. 2021On the Pitfalls of Analyzing Individual Neurons in Language Modelscited by 62
  21. 2018Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Taskscited by 56
  22. 2022Discovering Latent Concepts Learned in BERTcited by 56
  23. 2020Intrinsic Probing through Dimension Selectioncited by 45
  24. 2018The Importance of Being Recurrent for Modeling Hierarchical Structurecited by 42
  25. 2016Representation of linguistic form and function in recurrent neural networkscited by 39
  26. 2022Analyzing Encoded Concepts in Transformer Language Modelscited by 38
  27. 2019On the Linguistic Representational Power of Neural Machine Translation Modelscited by 34
  28. 2020Asking without Telling: Exploring Latent Ontologies in Contextual Representationscited by 34
  29. 2021How transfer learning impacts linguistic knowledge in deep NLP models?cited by 32
  30. 2016Word Representation Models for Morphologically Rich Languages in Neural Machine Translationcited by 9
  31. 2022Latent Topology Induction for Understanding Contextualized Representationscited by 7
  32. 2022ConceptX: A Framework for Latent Concept Analysiscited by 6
  33. 2022Identifying and Measuring Token-Level Sentiment Bias in Pre-trained Language Models with Promptscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 6; 2026: 3.

16 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  2. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 235
  3. 2026There Is More to Refusal in Large Language Models than a Single Directioncited by 26
  4. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 25
  5. 2024Latent Concept-based Explanation of NLP Modelscited by 13
  6. 2024Exploring Alignment in Shared Cross-lingual Spacescited by 13
  7. 2025Do Activation Verbalization Methods Convey Privileged Information?cited by 10
  8. 2023Scaling up Discovery of Latent Concepts in Deep NLP Modelscited by 9
  9. 2024How Do Artificial Intelligences Think? The Three Mathematico-Cognitive Factors of Categorical Segmentation Operated by Synthetic Neuronscited by 3
  10. 2025Synthetic Categorical Restructuring large Or How AIs Gradually Extract Efficient Regularities from Their Experience of the Worldcited by 2
  11. 2025Beyond the Leaderboard: Understanding Performance Disparities in Large Language Models via Model Diffingcited by 2
  12. 2025From Words to Waves: Analyzing Concept Formation in Speech and Text-Based Foundation Modelscited by 1
4 more
  1. 2026Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025cited by 1
  2. 2025Intra-neuronal attention within language models Relationships between activation and semanticscited by 0
  3. 2025Polysemy of Synthetic Neurons Towards a New Type of Explanatory Categorical Vector Spacescited by 0
  4. 2026TRACE: Temporal Risk Assessment and Concept Editing for Safer Language Generationcited by 0