mlsanityData to 14 Oct 2026

Relational World Knowledge Representation in Contextual Language Models: A Review

arXiv 2104.05837 ↗First posted 12 Apr 2021cs.CL cs.AI cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Relational knowledge bases (KBs) are commonly used to represent world knowledge in machines. However, while advantageous for their high degree of precision and interpretability, KBs are usually organized according to manually-defined schemas, which limit their expressiveness and require significant human efforts to engineer and maintain. In this review, we take a natural language processing perspective to these limitations, examining how they may be addressed in part by training deep contextual language models (LMs) to internalize and express relational knowledge in more flexible forms. We propose to organize knowledge representation strategies in LMs by the level of KB supervision provided, from no KB supervision at all to entity- and relation-level supervision. Our contributions are threefold: (1) We provide a high-level, extensible taxonomy for knowledge representation in LMs; (2) Within our taxonomy, we highlight notable models, evaluation tasks, and findings, in order to provide an up-to-date review of current knowledge representation capabilities in LMs; and (3) We suggest future research directions that build upon the complementary aspects of LMs and KBs as knowledge representations.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 82 of its 130 references that are in the map. First posted on arXiv 12 Apr 2021.

Builds on

82 of 130 references.

References · 82

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  11. 2018Deep contextualized word representationscited by 2,290
  12. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
38 more
  1. 2019Language Models as Knowledge Bases?cited by 1,936
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  3. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
  4. 2019How Can We Know What Language Models Know?cited by 959
  5. 2019RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Spacecited by 898
  6. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  7. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  8. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
  9. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 528
  10. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  11. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  12. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  13. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  14. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  15. 2020Autoregressive Entity Retrievalcited by 334
  16. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  17. 2019KG-BERT: BERT for Knowledge Graph Completioncited by 324
  18. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 319
  19. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 304
  20. 2019Knowledge Enhanced Contextual Word Representationscited by 296
  21. 2019Matching the Blanks: Distributional Similarity for Relation Learningcited by 272
  22. 2020Transformers as Soft Reasoners over Languagecited by 270
  23. 2020COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphscited by 254
  24. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 243
  25. 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 231
  26. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  27. 2020How Context Affects Language Models' Factual Predictionscited by 181
  28. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  29. 2020Unsupervised Commonsense Question Answering with Self-Talkcited by 161
  30. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 138
  31. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  32. 2019Zero-Shot Entity Linking by Reading Entity Descriptionscited by 124
  33. 2020A Survey of Knowledge-Enhanced Text Generationcited by 120
  34. 2018Construction of the Literature Graph in Semantic Scholarcited by 118
  35. 2020Investigating Pretrained Language Models for Graph-to-Text Generationcited by 117
  36. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
  37. 2020A Knowledge-Enhanced Pretraining Model for Commonsense Story Generationcited by 116
  38. 2020Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Modelscited by 110

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 6; 2023: 14; 2024: 8; 2025: 1.

31 papers.

Sort:

By area:

  1. 2022Mass-Editing Memory in a Transformercited by 1,001
  2. 2022A Review on Language Models as Knowledge Basescited by 157
  3. 2023Large Language Models for Robotics: A Surveycited by 133
  4. 2022A Decade of Knowledge Graphs in Natural Language Processing: A Surveycited by 40
  5. 2021Knowledge Enhanced Pretrained Language Models: A Compreshensive Surveycited by 33
  6. 2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
  7. 2023An In-depth Survey of Large Language Model-based Artificial Intelligence Agentscited by 31
  8. 2023Are Large Language Models Geospatially Knowledgeable?cited by 30
  9. 2025Conformal Linguistic Calibration: Trading-off between Factuality and Specificitycited by 20
  10. 2024The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Modelscited by 17
  11. 2023Discovering Knowledge-Critical Subnetworks in Pretrained Language Modelscited by 16
  12. 2024Visual Knowledge in the Big Model Era: Retrospect and Prospectcited by 16
19 more
  1. 2024Core: Robust Factual Precision with Informative Sub-Claim Identificationcited by 16
  2. 2022COMPS: Conceptual Minimal Pair Sentences for testing Robust Property Knowledge and its Inheritance in Pre-trained Language Modelscited by 15
  3. 2024Automating the Enterprise with Foundation Modelscited by 14
  4. 2023Understanding Finetuning for Factual Knowledge Extraction from Language Modelscited by 13
  5. 2023Do PLMs Know and Understand Ontological Knowledge?cited by 13
  6. 2023Rethinking Language Models as Symbolic Knowledge Graphscited by 12
  7. 2022A survey on knowledge-enhanced multimodal learningcited by 11
  8. 2024A Hybrid RAG System with Comprehensive Enhancement on Complex Reasoningcited by 8
  9. 2024AGENTiGraph: An Interactive Knowledge Graph Platform for LLM-based Chatbots Utilizing Private Datacited by 8
  10. 2023Causal interventions expose implicit situation models for commonsense language understandingcited by 7
  11. 2023Exploring the Landscape of Natural Language Processing Researchcited by 7
  12. 2023NNKGC: Improving Knowledge Graph Completion with Node Neighborhoodscited by 5
  13. 2022A Survey on Knowledge-Enhanced Pre-trained Language Modelscited by 3
  14. 2023Domain-Expanded ASTE: Rethinking Generalization in Aspect Sentiment Triplet Extractioncited by 3
  15. 2024The Impact of Generative Artificial Intelligence on Ideation and the performance of Innovation Teams (Preprint)cited by 3
  16. 2021Commonsense Knowledge in Word Associations and ConceptNetcited by 2
  17. 2023Counteracts: Testing Stereotypical Representation in Pre-trained Language Modelscited by 1
  18. 2024Synergizing Machine Learning & Symbolic Methods: A Survey on Hybrid Approaches to Natural Language Processingcited by 1
  19. 2023KEPLET: Knowledge-Enhanced Pretrained Language Model with Topic Entity Awarenesscited by 0