Data to 13 Oct 2026

Self-Alignment Pretraining for Biomedical Entity Representations

arXiv 2010.11784 ↗First posted 22 Oct 2020cs.CL cs.AI cs.LGcited by 234

Not on the map. Why?

Abstract (arXiv)

Despite the widespread success of self-supervised learning via masked language models (MLM), accurately capturing fine-grained semantic relationships in the biomedical domain remains a challenge. This is of paramount importance for entity-level tasks such as entity linking where the ability to model entity relations (especially synonymy) is pivotal. To address this challenge, we propose SapBERT, a pretraining scheme that self-aligns the representation space of biomedical entities. We design a scalable metric learning framework that can leverage UMLS, a massive collection of biomedical ontologies with 4M+ concepts. In contrast with previous pipeline-based hybrid systems, SapBERT offers an elegant one-model-for-all solution to the problem of medical entity linking (MEL), achieving a new state-of-the-art (SOTA) on six MEL benchmarking datasets. In the scientific domain, we achieve SOTA even without task-specific supervision. With substantial improvement over various domain-specific pretrained MLMs such as BioBERT, SciBERTand and PubMedBERT, our pretraining scheme proves to be both effective and robust.

Key facts. Cited by 234 papers in the map (data through 13 Oct 2026). First posted on arXiv 22 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 23; 2022: 32; 2023: 48; 2024: 38; 2025: 39; 2026: 54.

234 papers.

Sort:

By area:

2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
2022GreaseLM: Graph REASoning Enhanced Language Models for Question Answeringcited by 170
2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
2022RoentGen: Vision-Language Foundation Model for Chest X-ray Generationcited by 89
2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
2022Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domainscited by 85
2022BioBART: Pretraining and Evaluation of A Biomedical Generative Language Modelcited by 83
2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
2023Matching Patients to Clinical Trials with Large Language Modelscited by 69

38 more

2022Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Codingcited by 40
2021Does the Magic of BERT Apply to Medical Code Assignment? A Quantitative Studycited by 39
2024BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrieverscited by 38
2023GeoLM: Empowering Language Models for Geospatially Grounded Language Understandingcited by 37
2023OLaLa: Ontology Matching with Large Language Modelscited by 35
2022Generative Biomedical Entity Linking via Knowledge Base-Guided Pre-training and Synonyms-Aware Fine-tuningcited by 34
2021AMMU : A Survey of Transformer-based Biomedical Pretrained Language Modelscited by 32
2021TaCL: Improving BERT Pre-training with Token-aware Contrastive Learningcited by 32
2021Knowledge-Rich Self-Supervision for Biomedical Entity Linkingcited by 31
2022Summarizing Patients Problems from Hospital Progress Notes Using Pre-trained Sequence-to-Sequence Modelscited by 30
2023Leveraging Medical Knowledge Graphs Into Large Language Models for Diagnosis Prediction: Design and Application Studycited by 30
2023KnowGPT: Knowledge Graph based Prompting for Large Language Modelscited by 30
2021Learning Domain-Specialised Representations for Cross-Lingual Biomedical Entity Linkingcited by 27
2024Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generationcited by 27
2024Language Models as Hierarchy Encoderscited by 26
2025Clinical ModernBERT: An efficient and long context encoder for biomedical textcited by 26
2021ConvFiT: Conversational Fine-Tuning of Pretrained Language Modelscited by 25
2023MKRAG: Medical Knowledge Retrieval Augmented Generation for Medical Question Answeringcited by 23
2021Entity Linking and Discovery via Arborescence-based Supervised Clusteringcited by 22
2021Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language Modelscited by 21
2022BioLORD: Learning Ontological Representations from Definitions (for Biomedical Concepts and their Textual Descriptions)cited by 20
2021MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Modelscited by 18
2022Improving Word Translation via Two-Stage Contrastive Learningcited by 18
2023Overview of BioASQ 2023: The eleventh BioASQ challenge on Large-Scale Biomedical Semantic Indexing and Question Answeringcited by 18
2023GrapeQA: GRaph Augmentation and Pruning to Enhance Question-Answeringcited by 17
2021Cross-Domain Data Integration for Named Entity Disambiguation in Biomedical Textcited by 15
2024PromptLink: Leveraging Large Language Models for Cross-Source Biomedical Concept Linkingcited by 15
2021BERT might be Overkill: A Tiny but Effective Biomedical Entity Linker based on Residual Convolutional Neural Networkscited by 14
2021Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERTcited by 14
2022DR.BENCH: Diagnostic Reasoning Benchmark for Clinical Natural Language Processingcited by 14
2023Vision-Language Generative Model for View-Specific Chest X-ray Generationcited by 14
2023A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarizationcited by 14
2023Evaluation of large language models for discovery of gene set functioncited by 14
2024Ontology Embedding: A Survey of Methods, Applications and Resourcescited by 12
2025Large Language Models with Temporal Reasoning for Longitudinal Clinical Summarization and Predictioncited by 12
2025KERAP: A Knowledge-Enhanced Reasoning Approach for Accurate Zero-shot Diagnosis Prediction Using Multi-agent LLMscited by 12
2022Detection, Disambiguation, Re-ranking: Autoregressive Entity Linking as a Multi-Task Problemcited by 11
2022Embracing Ambiguity: Improving Similarity-oriented Tasks with Contextual Synonym Knowledgecited by 11

50 most cited of 234 All 234 (JSON)

Similar, not cited

Close in wording, no citation either way