mlsanityData to 13 Oct 2026

BioLORD-2023: Semantic Textual Representations Fusing LLM and Clinical Knowledge Graph Insights

arXiv 2311.16075 ↗First posted 27 Nov 2023cs.CL cs.AI cs.IRcited by 7

Not on the map. Why?

Abstract (arXiv)

In this study, we investigate the potential of Large Language Models to complement biomedical knowledge graphs in the training of semantic models for the biomedical and clinical domains. Drawing on the wealth of the UMLS knowledge graph and harnessing cutting-edge Large Language Models, we propose a new state-of-the-art approach for obtaining high-fidelity representations of biomedical concepts and sentences, consisting of three steps: an improved contrastive learning phase, a novel self-distillation phase, and a weight averaging phase. Through rigorous evaluations via the extensive BioLORD testing suite and diverse downstream tasks, we demonstrate consistent and substantial performance improvements over the previous state of the art (e.g. +2pts on MedSTS, +2.5pts on MedNLI-S, +6.1pts on EHR-Rel-B). Besides our new state-of-the-art biomedical model for English, we also distill and release a multilingual model compatible with 50+ languages and finetuned on 7 European languages. Many clinical pipelines can benefit from our latest models. Our new multilingual model enables a range of languages to benefit from our advancements in biomedical semantic representation learning, opening a new avenue for bioinformatics researchers around the world. As a result, we hope to see BioLORD-2023 becoming a precious tool for future biomedical applications.

Key facts. Cited by 7 papers in the map (data through 13 Oct 2026). Builds on 39 of its 59 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

39 of 59 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2023GPT-4 Technical Reportcited by 18,278
  3. 2022Training language models to follow instructions with human feedbackcited by 15,856
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,221
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,107
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,712
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,170
  8. 2022Large Language Models Encode Clinical Knowledgecited by 1,817
  9. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,384
  10. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  11. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 774
  12. 2022Galactica: A Large Language Model for Sciencecited by 741
27 more
  1. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 666
  2. 2020Language-agnostic BERT Sentence Embeddingcited by 622
  3. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  4. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 536
  5. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
  6. 2020Self-Alignment Pretraining for Biomedical Entity Representationscited by 234
  7. 2018Lessons from Natural Language Inference in the Clinical Domaincited by 150
  8. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  9. 2023ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluationcited by 87
  10. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  11. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  12. 2023Matching Patients to Clinical Trials with Large Language Modelscited by 69
  13. 2020Biomedical Entity Representations with Synonym Marginalizationcited by 59
  14. 2018MedSTS: A Resource for Clinical Semantic Textual Similaritycited by 39
  15. 2023Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4Vcited by 38
  16. 2023Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applicationscited by 28
  17. 2021Learning Domain-Specialised Representations for Cross-Lingual Biomedical Entity Linkingcited by 27
  18. 2022BioLORD: Learning Ontological Representations from Definitions (for Biomedical Concepts and their Textual Descriptions)cited by 20
  19. 2023A Review on Knowledge Graphs for Healthcare: Resources, Applications, and Promisescited by 13
  20. 2023Comprehensive evaluation of deep and graph learning on drug-drug interactions predictioncited by 10
  21. 2022Generalizing over Long Tail Concepts for Medical Term Normalizationcited by 8
  22. 2019Multi-task Learning and Catastrophic Forgetting in Continual Reinforcement Learningcited by 5
  23. 2021A Hybrid Approach to Measure Semantic Relatedness in Biomedical Conceptscited by 5
  24. 2023Automatic Glossary of Clinical Terminology: a Large-Scale Dictionary of Biomedical Definitions Generated from Ontological Knowledgecited by 5
  25. 2020Biomedical Concept Relatedness -- A large EHR-based benchmarkcited by 4
  26. 2023Boosting Adverse Drug Event Normalization on Social Media: General-Purpose Model Initialization and Biomedical Semantic Text Similarity Benefit Zero-Shot Linking in Informal Contextscited by 2
  27. 2023Navigating Healthcare Insights: A Birds Eye View of Explainability with Knowledge Graphscited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 2.

7 papers.

  1. 2024MedSyn: LLM-based Synthetic Medical Text Generation Frameworkcited by 19
  2. 2025BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domaincited by 10
  3. 2024From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practicecited by 8
  4. 2024Clinical Reasoning over Tabular Data and Text with Bayesian Networkscited by 4
  5. 2025Modeling Clinical Uncertainty in Radiology Reports: from Explicit Uncertainty Markers to Implicit Reasoning Pathwayscited by 4
  6. 2026Language corpora for the Dutch medical domaincited by 0
  7. 2026MMed-Bench-IR: A Heterogeneous Benchmark for Multilingual Medical Information Retrievalcited by 0