mlsanityData to 14 Oct 2026

Inspecting the concept knowledge graph encoded by modern language models

arXiv 2105.13471 ↗First posted 27 May 2021cs.AI cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

The field of natural language understanding has experienced exponential progress in the last few years, with impressive results in several tasks. This success has motivated researchers to study the underlying knowledge encoded by these models. Despite this, attempts to understand their semantic capabilities have not been successful, often leading to non-conclusive, or contradictory conclusions among different works. Via a probing classifier, we extract the underlying knowledge graph of nine of the most influential language models of the last years, including word embeddings, text generators, and context encoders. This probe is based on concept relatedness, grounded on WordNet. Our results reveal that all the models encode this knowledge, but suffer from several inaccuracies. Furthermore, we show that the different architectures and training strategies lead to different model biases. We conduct a systematic evaluation to discover specific factors that explain why some concepts are challenging. We hope our insights will motivate the development of models that capture concepts more precisely.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 48 of its 74 references that are in the map. First posted on arXiv 27 May 2021.

Builds on

48 of 74 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  9. 2018Deep contextualized word representationscited by 2,290
  10. 2019Language Models as Knowledge Bases?cited by 1,936
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  12. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
36 more
  1. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  2. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  3. 2019Are Sixteen Heads Really Better than One?cited by 816
  4. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  5. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  6. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  7. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  8. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  9. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
  10. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 320
  11. 2019Revealing the Dark Secrets of BERTcited by 301
  12. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  13. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 273
  14. 2019Matching the Blanks: Distributional Similarity for Relation Learningcited by 272
  15. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  16. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 226
  17. 2019Visualizing and Measuring the Geometry of BERTcited by 216
  18. 2019Assessing BERT's Syntactic Abilitiescited by 192
  19. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  20. 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 151
  21. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
  22. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  23. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 126
  24. 2017Building machines that adapt and compute like brainscited by 99
  25. 2019Visualizing and Understanding the Effectiveness of BERTcited by 67
  26. 2020Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformerscited by 67
  27. 2019Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddingscited by 56
  28. 2019Improving Transformer Models by Reordering their Sublayerscited by 52
  29. 2015Relation Classification via Recurrent Neural Networkcited by 46
  30. 2020A Frustratingly Easy Approach for Entity and Relation Extractioncited by 37
  31. 2019What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?cited by 30
  32. 2020Controlling Computation versus Quality for Neural Sequence Modelscited by 27
  33. 2018N-ary Relation Extraction using Graph State LSTMcited by 26
  34. 2020What's in a Name? Are BERT Named Entity Representations just as Good for any other Name?cited by 17
  35. 2018An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translationcited by 15
  36. 2019Probing for Semantic Classes: Diagnosing the Meaning Content of Word Embeddingscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 3; 2023: 3; 2024: 2; 2025: 2; 2026: 3.

15 papers.

Sort:

By area:

  1. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  2. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  3. 2022COPEN: Probing Conceptual Knowledge in Pre-trained Language Modelscited by 21
  4. 2021Not All Models Localize Linguistic Knowledge in the Same Place: A Layer-wise Probing on BERToids' Representationscited by 16
  5. 2024Towards a theory of model distillationcited by 15
  6. 2025Self-Guided Process Reward Optimization with Redefined Step-wise Advantage for Process Reinforcement Learningcited by 11
  7. 2023Language-Specific Representation of Emotion-Concept Knowledge Causally Supports Emotion Inferencecited by 6
  8. 2021Augmenting BERT-style Models with Predictive Coding to Improve Discourse-level Representationscited by 5
  9. 2022PALT: Parameter-Lite Transfer of Language Models for Knowledge Graph Completioncited by 3
  10. 2026The Lattice Representation Hypothesis of Large Language Modelscited by 3
  11. 2023Optimizing Factual Accuracy in Text Generation through Dynamic Knowledge Selectioncited by 2
  12. 2025From Tokens to Lattices: Emergent Lattice Structures in Language Modelscited by 2
3 more
  1. 2026Do Language Models Encode Semantic Relations? Probing and Sparse Feature Analysiscited by 2
  2. 2024ConcEPT: Concept-Enhanced Pre-Training for Language Modelscited by 1
  3. 2026Relation Geometry in Semantic Space of Language Modelscited by 1