mlsanityData to 7 Oct 2026

Combining pre-trained language models and structured knowledge

arXiv 2101.12294 ↗First posted 28 Jan 2021cs.CLcited by 25

Not on the map. Why?

Abstract (arXiv)

In recent years, transformer-based language models have achieved state of the art performance in various NLP benchmarks. These models are able to extract mostly distributional information with some semantics from unstructured text, however it has proven challenging to integrate structured information, such as knowledge graphs into these models. We examine a variety of approaches to integrate structured knowledge into current language models and determine challenges, and possible opportunities to leverage both structured and unstructured information sources. From our survey, we find that there are still opportunities at exploiting adapter-based injections and that it may be possible to further combine various of the explored approaches into one system.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 29 of its 45 references that are in the map. First posted on arXiv 28 Jan 2021.

Builds on

29 of 45 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2019Language Models as Knowledge Bases?cited by 1,922
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  9. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  10. 2020Knowledge Graphscited by 619
  11. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  12. 2019KG-BERT: BERT for Knowledge Graph Completioncited by 323
17 more
  1. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  2. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 303
  3. 2019Knowledge Enhanced Contextual Word Representationscited by 294
  4. 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 230
  5. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  6. 2019Semantics-aware BERT for Language Understandingcited by 111
  7. 2019Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question Answeringcited by 68
  8. 2019Commonsense Knowledge Base Completion with Structural and Semantic Contextcited by 61
  9. 2020Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformerscited by 57
  10. 2020Knowledge-Aware Language Model Pretrainingcited by 52
  11. 2019Metalearned Neural Memorycited by 44
  12. 2019Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Modelscited by 28
  13. 2019Integrating Graph Contextualized Knowledge into Pre-trained Language Modelscited by 26
  14. 2020Exploiting Structured Knowledge in Text via Graph-Guided Representation Learningcited by 26
  15. 2019Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representationscited by 24
  16. 2019MCScript2.0: A Machine Comprehension Corpus Focused on Script Events and Participantscited by 20
  17. 2019Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-shot Commonsense Question Answeringcited by 16

Built on by

Cited by year

Citing papers per year: 2021: 6; 2022: 6; 2023: 7; 2024: 6.

25 papers.

Sort:

By area:

  1. 2022A Review on Language Models as Knowledge Basescited by 156
  2. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  3. 2023Structured prompt interrogation and recursive extraction of semantics (SPIRES): A method for populating knowledge bases using zero-shot learningcited by 50
  4. 2022A Decade of Knowledge Graphs in Natural Language Processing: A Surveycited by 40
  5. 2021Knowledge Enhanced Pretrained Language Models: A Compreshensive Surveycited by 33
  6. 2021Relational World Knowledge Representation in Contextual Language Models: A Reviewcited by 31
  7. 2021DKPLM: Decomposable Knowledge-enhanced Pre-trained Language Model for Natural Language Understandingcited by 15
  8. 2023Wikiformer: Pre-training with Structured Information of Wikipedia for Ad-hoc Retrievalcited by 15
  9. 2024Lost-in-Distance: Impact of Contextual Proximity on LLM Performance in Graph Taskscited by 8
  10. 2023cTBLS: Augmenting Large Language Models with Conversational Tablescited by 7
  11. 2023Causal interventions expose implicit situation models for commonsense language understandingcited by 7
  12. 2021Multilingual Fact Linkingcited by 6
13 more
  1. 2022Revisiting and Advancing Chinese Natural Language Understanding with Accelerated Heterogeneous Knowledge Pre-trainingcited by 6
  2. 2022Injecting Domain Knowledge in Language Models for Task-Oriented Dialogue Systemscited by 6
  3. 2022Transformers and the representation of biomedical background knowledgecited by 5
  4. 2023Improving (Dis)agreement Detection with Inductive Social Relation Information From Comment-Reply Interactionscited by 5
  5. 2021Should Semantic Vector Composition be Explicit? Can it be Linear?cited by 3
  6. 2022A Survey on Knowledge-Enhanced Pre-trained Language Modelscited by 3
  7. 2024Reasoning Factual Knowledge in Structured Data with Large Language Modelscited by 3
  8. 2023Diversifying Knowledge Enhancement of Biomedical Language Models using Adapter Modules and Knowledge Graphscited by 1
  9. 2023Profiling the news spreading barriers using news headlinescited by 0
  10. 2024TRELM: Towards Robust and Efficient Pre-training for Knowledge-Enhanced Language Modelscited by 0
  11. 2024Empowering Small-Scale Knowledge Graphs: A Strategy of Leveraging General-Purpose Knowledge Graphs for Enriched Embeddingscited by 0
  12. 2024YouTube Comments Decoded: Leveraging LLMs for Low Resource Language Classificationcited by 0
  13. 2024Adapter-based Approaches to Knowledge-enhanced Language Models -- A Surveycited by 0