mlsanityData to 7 Oct 2026

Deep Bidirectional Language-Knowledge Graph Pretraining

arXiv 2210.09338 ↗First posted 17 Oct 2022cs.CL cs.AI cs.LGcited by 156

Not on the map. Why?

Abstract (arXiv)

Pretraining a language model (LM) on text has been shown to help various downstream NLP tasks. Recent works show that a knowledge graph (KG) can complement text data, offering structured background knowledge that provides a useful scaffold for reasoning. However, these works are not pretrained to learn a deep fusion of the two modalities at scale, limiting the potential to acquire fully joint representations of text and KG. Here we propose DRAGON (Deep Bidirectional Language-Knowledge Graph Pretraining), a self-supervised approach to pretraining a deeply joint language-knowledge foundation model from text and KG at scale. Specifically, our model takes pairs of text segments and relevant KG subgraphs as input and bidirectionally fuses information from both modalities. We pretrain this model by unifying two self-supervised reasoning tasks, masked language modeling and KG link prediction. DRAGON outperforms existing LM and LM+KG models on diverse downstream tasks including question answering across general and biomedical domains, with +5% absolute gain on average. In particular, DRAGON achieves notable performance on complex reasoning about language and knowledge (+10% on questions involving long contexts or multi-step reasoning) and low-resource QA (+8% on OBQA and RiddleSense), and new state-of-the-art results on various BioNLP tasks. Our code and trained models are available at https://github.com/michiyasunaga/dragon.

Key facts. Cited by 156 papers in the map (data through 7 Oct 2026). Builds on 65 of its 88 references that are in the map. First posted on arXiv 17 Oct 2022.

Builds on

65 of 88 references.

References · 65

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2018Deep contextualized word representationscited by 2,285
  12. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
38 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  3. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  4. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  5. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  6. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  7. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  8. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  10. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  11. 2021Ethical and social risks of harm from Language Modelscited by 957
  12. 2014Embedding Entities and Relations for Learning and Inference in Knowledge Basescited by 929
  13. 2016Complex Embeddings for Simple Link Predictioncited by 897
  14. 2019RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Spacecited by 893
  15. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  16. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  17. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  18. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  19. 2019KG-BERT: BERT for Knowledge Graph Completioncited by 323
  20. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 303
  21. 2019Abductive Commonsense Reasoningcited by 300
  22. 2019Knowledge Enhanced Contextual Word Representationscited by 294
  23. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  24. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  25. 2018Modeling polypharmacy side effects with graph convolutional networkscited by 237
  26. 2020Self-Alignment Pretraining for Biomedical Entity Representationscited by 233
  27. 2019KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoningcited by 216
  28. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 209
  29. 2018SimplE Embedding for Link Prediction in Knowledge Graphscited by 195
  30. 2019PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Textcited by 182
  31. 2022GreaseLM: Graph REASoning Enhanced Language Models for Question Answeringcited by 170
  32. 2020Query2box: Reasoning over Knowledge Graphs in Vector Space using Box Embeddingscited by 169
  33. 2018Open Domain Question Answering Using Early Fusion of Knowledge Bases and Textcited by 167
  34. 2020Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answeringcited by 122
  35. 2020A Survey of Knowledge-Enhanced Text Generationcited by 119
  36. 2020Towards Controllable Biases in Language Generationcited by 112
  37. 2019Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Modelcited by 102
  38. 2020CoLAKE: Contextualized Language and Knowledge Embeddingcited by 86

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 57; 2024: 56; 2025: 27; 2026: 10.

156 papers.

Sort:

By area:

  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  2. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  3. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  4. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  5. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  6. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  7. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  8. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  9. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271
  10. 2023Challenges and Applications of Large Language Modelscited by 251
  11. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 190
  12. 2023Talk like a Graph: Encoding Graphs for Large Language Modelscited by 175
38 more
  1. 2023Knowledge Graph Prompting for Multi-Document Question Answeringcited by 150
  2. 2024MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoningcited by 145
  3. 2023Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learningcited by 132
  4. 2024GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoningcited by 128
  5. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  6. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  7. 2023Almanac: Retrieval-Augmented Language Models for Clinical Medicinecited by 122
  8. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  9. 2023Large Language Models as Analogical Reasonerscited by 107
  10. 2022A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multimodalcited by 106
  11. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  12. 2024Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generationcited by 89
  13. 2023On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligencecited by 88
  14. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 85
  15. 2024Position: Graph Foundation Models are Already Herecited by 84
  16. 2023Benchmarking large language models for biomedical natural language processing applications and recommendationscited by 78
  17. 2023MindMap: Knowledge Graph Prompting Sparks Graph of Thoughts in Large Language Modelscited by 73
  18. 2023Graph Neural Prompting with Large Language Modelscited by 64
  19. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  20. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  21. 2023FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledgecited by 55
  22. 2024Text-space Graph Foundation Models: Comprehensive Benchmarks and New Insightscited by 44
  23. 2024Multimodal Reasoning with Multimodal Knowledge Graphcited by 43
  24. 2025Self-Evolving Multi-Agent Simulations for Realistic Clinical Interactionscited by 40
  25. 2023Knowledge Solver: Teaching LLMs to Search for Domain Knowledge from Knowledge Graphscited by 35
  26. 2023Graph Reasoning for Question Answering with Triplet Retrievalcited by 34
  27. 2023Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applicationscited by 33
  28. 2024Graph Machine Learning in the Era of Large Language Models (LLMs)cited by 33
  29. 2024JMLR: Joint Medical LLM and Retrieval Training for Enhancing Reasoning and Professional Question Answering Capabilitycited by 32
  30. 2023Biomedical knowledge graph-optimized prompt generation for large language modelscited by 30
  31. 2023KnowGPT: Knowledge Graph based Prompting for Large Language Modelscited by 30
  32. 2025Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunitiescited by 30
  33. 2023IGB: Addressing The Gaps In Labeling, Features, Heterogeneity, and Size of Public Graph Datasets for Deep Learning Researchcited by 29
  34. 2023Train Your Own GNN Teacher: Graph-Aware Distillation on Textual Graphscited by 29
  35. 2023Leveraging Medical Knowledge Graphs Into Large Language Models for Diagnosis Prediction: Design and Application Studycited by 29
  36. 2023Large Language Models as Topological Structure Enhancers for Text-Attributed Graphscited by 27
  37. 2024Question Calibration and Multi-Hop Modeling for Temporal Question Answeringcited by 25
  38. 2023Structuring Representation Geometry with Rotationally Equivariant Contrastive Learningcited by 24

50 most cited of 156 All 156 (JSON)