mlsanityData to 7 Oct 2026

LinkBERT: Pretraining Language Models with Document Links

arXiv 2203.15827 ↗First posted 29 Mar 2022cs.CL cs.LGcited by 209

Not on the map. Why?

Abstract (arXiv)

Language model (LM) pretraining can learn various knowledge from text corpora, helping downstream tasks. However, existing methods such as BERT model a single document, and do not capture dependencies or knowledge that span across documents. In this work, we propose LinkBERT, an LM pretraining method that leverages links between documents, e.g., hyperlinks. Given a text corpus, we view it as a graph of documents and create LM inputs by placing linked documents in the same context. We then pretrain the LM with two joint self-supervised objectives: masked language modeling and our new proposal, document relation prediction. We show that LinkBERT outperforms BERT on various downstream tasks across two domains: the general domain (pretrained on Wikipedia with hyperlinks) and biomedical domain (pretrained on PubMed with citation links). LinkBERT is especially effective for multi-hop reasoning and few-shot QA (+5% absolute improvement on HotpotQA and TriviaQA), and our biomedical LinkBERT sets new states of the art on various BioNLP tasks (+7% on BioASQ and USMLE). We release our pretrained models, LinkBERT and BioLinkBERT, as well as code and data at https://github.com/michiyasunaga/LinkBERT.

Key facts. Cited by 209 papers in the map (data through 7 Oct 2026). Builds on 56 of its 98 references that are in the map. First posted on arXiv 29 Mar 2022.

Builds on

56 of 98 references.

References · 56

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  11. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  12. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
38 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  4. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  5. 2019Language Models as Knowledge Bases?cited by 1,922
  6. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  7. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  8. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  9. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  10. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  11. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  12. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  13. 2019Strategies for Pre-training Graph Neural Networkscited by 964
  14. 2018Neural Network Acceptability Judgmentscited by 952
  15. 2014Embedding Entities and Relations for Learning and Inference in Knowledge Basescited by 929
  16. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  17. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  18. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  19. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  20. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  21. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
  22. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  23. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  24. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  25. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  26. 2017SearchQA: A New Q&A Dataset Augmented with Context from a Search Enginecited by 221
  27. 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 183
  28. 2022GreaseLM: Graph REASoning Enhanced Language Models for Question Answeringcited by 170
  29. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  30. 2019Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answeringcited by 147
  31. 2020Towards Controllable Biases in Language Generationcited by 112
  32. 2020CoLAKE: Contextualized Language and Knowledge Embeddingcited by 86
  33. 2018A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literaturecited by 79
  34. 2019ScisummNet: A Large Annotated Corpus and Content-Impact Models for Scientific Paper Summarization with Citation Networkscited by 73
  35. 2020Relational Message Passing for Knowledge Graph Completioncited by 61
  36. 2021HTLM: Hyper-Text Pre-Training and Prompting of Language Modelscited by 58
  37. 2017Graph-based Neural Multi-Document Summarizationcited by 54
  38. 2018Content-Based Citation Recommendationcited by 43

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 26; 2023: 69; 2024: 60; 2025: 35; 2026: 19.

209 papers.

Sort:

By area:

  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  2. 2022Galactica: A Large Language Model for Sciencecited by 736
  3. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  4. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  5. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  6. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  7. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271
  8. 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
  9. 2022Can large language models reason about medical questions?cited by 194
  10. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 190
  11. 2024Retrieval-Augmented Generation with Graphs (GraphRAG)cited by 170
  12. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
38 more
  1. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  2. 2022Learning on Large-scale Text-attributed Graphs via Variational Inferencecited by 153
  3. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  4. 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 124
  5. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  6. 2023Large Language Models as Analogical Reasonerscited by 107
  7. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  8. 2024Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generationcited by 89
  9. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 85
  10. 2024Scientific Large Language Models: A Survey on Biological & Chemical Domainscited by 80
  11. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 76
  12. 2023Matching Patients to Clinical Trials with Large Language Modelscited by 68
  13. 2023Graph Neural Prompting with Large Language Modelscited by 64
  14. 2023Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Taskscited by 62
  15. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 62
  16. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  17. 2023MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Recordscited by 54
  18. 2023Patton: Language Model Pretraining on Text-Rich Networkscited by 49
  19. 2023Text-guided Foundation Model Adaptation for Pathological Image Classificationcited by 47
  20. 2022TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twittercited by 44
  21. 2024Text-space Graph Foundation Models: Comprehensive Benchmarks and New Insightscited by 44
  22. 2025Self-Evolving Multi-Agent Simulations for Realistic Clinical Interactionscited by 40
  23. 2022BigBIO: A Framework for Data-Centric Biomedical Natural Language Processingcited by 36
  24. 2025Neuro-Symbolic AI in 2024: A Systematic Reviewcited by 36
  25. 2023ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviewscited by 34
  26. 2023ConGraT: Self-Supervised Contrastive Pretraining for Joint Graph and Text Embeddingscited by 32
  27. 2024JMLR: Joint Medical LLM and Retrieval Training for Enhancing Reasoning and Professional Question Answering Capabilitycited by 32
  28. 2023Train Your Own GNN Teacher: Graph-Aware Distillation on Textual Graphscited by 29
  29. 2023Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answeringcited by 28
  30. 2023GRENADE: Graph-Centric Language Model for Self-Supervised Representation Learning on Text-Attributed Graphscited by 28
  31. 2024Data-Centric Foundation Models in Computational Healthcare: A Surveycited by 27
  32. 2023nach0: Multimodal Natural and Chemical Languages Foundation Modelcited by 26
  33. 2024DTGB: A Comprehensive Benchmark for Dynamic Text-Attributed Graphscited by 26
  34. 2024ECG Semantic Integrator (ESI): A Foundation ECG Model Pretrained with LLM-Enhanced Cardiological Textcited by 25
  35. 2022Optimizing Bi-Encoder for Named Entity Recognition via Contrastive Learningcited by 24
  36. 2024RAG-RLRC-LaySum at BioLaySumm: Integrating Retrieval-Augmented Generation and Readability Control for Layman Summarization of Biomedical Textscited by 23
  37. 2022A Computational Inflection for Scientific Discoverycited by 22
  38. 2023Large language models in medicine: the potentials and pitfallscited by 21

50 most cited of 209 All 209 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Self-Alignment Pretraining for Biomedical Entity Representations
  2. 2024Synthetic continued pretraining
  3. 2026Injecting Structured Biomedical Knowledge into Language Models: Continual Pretraining vs. GraphRAG
  4. 2025Synthetic bootstrapped pretraining
  5. 2024Pre-training and Diagnosing Knowledge Base Completion Models
  6. 2023Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrieval
  7. 2025Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applications
  8. 2025Thinking Augmented Pre-training