mlsanityData to 7 Oct 2026

MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval

arXiv 2307.00589 ↗First posted 2 Jul 2023cs.IR cs.AI cs.CLcited by 182

Not on the map. Why?

Cited by 48 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Information retrieval (IR) is essential in biomedical knowledge acquisition and clinical decision support. While recent progress has shown that language model encoders perform better semantic retrieval, training such models requires abundant query-article annotations that are difficult to obtain in biomedicine. As a result, most biomedical IR systems only conduct lexical matching. In response, we introduce MedCPT, a first-of-its-kind Contrastively Pre-trained Transformer model for zero-shot semantic IR in biomedicine. To train MedCPT, we collected an unprecedented scale of 255 million user click logs from PubMed. With such data, we use contrastive learning to train a pair of closely-integrated retriever and re-ranker. Experimental results show that MedCPT sets new state-of-the-art performance on six biomedical IR tasks, outperforming various baselines including much larger models such as GPT-3-sized cpt-text-XL. In addition, MedCPT also generates better biomedical article and sentence representations for semantic evaluations. As such, MedCPT can be readily applied to various real-world biomedical IR tasks.

Key facts. Cited by 182 papers in the map (data through 7 Oct 2026). Builds on 43 of its 84 references that are in the map. First posted on arXiv 2 Jul 2023.

Builds on

43 of 84 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  9. 2017Billion-scale similarity search with GPUscited by 2,477
  10. 2018Deep contextualized word representationscited by 2,285
  11. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  12. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
31 more
  1. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  2. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  3. 2019Simplifying Graph Convolutional Networkscited by 1,470
  4. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  5. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  6. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  7. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  8. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  9. 2019Publicly Available Clinical BERT Embeddingscited by 995
  10. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  11. 2014Distributed Representations of Sentences and Documentscited by 677
  12. 2019Passage Re-ranking with BERTcited by 629
  13. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  14. 2018Universal Sentence Encodercited by 485
  15. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  16. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  17. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
  18. 2023Augmented Language Models: a Surveycited by 398
  19. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  20. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  21. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  22. 2020CORD-19: The COVID-19 Open Research Datasetcited by 179
  23. 2019A Question-Entailment Approach to Question Answeringcited by 133
  24. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  25. 2017Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Featurescited by 103
  26. 2022Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddingscited by 75
  27. 2021Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipelinecited by 65
  28. 2018BioSentVec: creating sentence embeddings for biomedical textscited by 47
  29. 2021Biomedical Question Answering: A Survey of Approaches and Challengescited by 46
  30. 2018Content-Based Citation Recommendationcited by 43
  31. 2018MedSTS: A Resource for Clinical Semantic Textual Similaritycited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 46; 2025: 64; 2026: 66.

182 papers.

Sort:

By area:

  1. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  2. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 212
  3. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  4. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  5. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  6. 2024MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicinecited by 79
  7. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  8. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 70
  9. 2023Matching Patients to Clinical Trials with Large Language Modelscited by 68
  10. 2024Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancementcited by 66
  11. 2024RaTEScore: A Metric for Radiology Report Generationcited by 65
  12. 2024Improving Retrieval-Augmented Generation in Medicine with Iterative Follow-up Questionscited by 54
38 more
  1. 2025An Agentic System for Rare Disease Diagnosis with Traceable Reasoningcited by 50
  2. 2024MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answeringcited by 43
  3. 2024Rationale-Guided Retrieval Augmented Generation for Medical Question Answeringcited by 41
  4. 2024BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrieverscited by 38
  5. 2025MedicalAgentsBench for Complex Medical Reasoning: Comparing Internalized Reasoning Models versus Externalized Agent-based Frameworkscited by 38
  6. 2024KG-Rank: Enhancing Large Language Models for Medical QA with Knowledge Graphs and Ranking Techniquescited by 37
  7. 2024Improving Medical Reasoning through Retrieval and Self-Reflection with Retrieval-Augmented Large Language Modelscited by 34
  8. 2024Mix-of-Granularity: Optimize the Chunking Granularity for Retrieval-Augmented Generationcited by 23
  9. 2024ColaCare: Enhancing Electronic Health Record Modeling through Large Language Model-Driven Multi-Agent Collaborationcited by 23
  10. 2024Aloe: A Family of Fine-tuned Open Healthcare LLMscited by 20
  11. 2024OLAPH: Improving Factuality in Biomedical Long-form Question Answeringcited by 20
  12. 2025Supervising the search process produces reliable and generalizable information-seeking agentscited by 20
  13. 2023Large-scale Long-tailed Disease Diagnosis on Radiology Imagescited by 17
  14. 2024A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industrycited by 17
  15. 2024Boosting Masked ECG-Text Auto-Encoders as Discriminative Learnerscited by 16
  16. 2024Accelerating Clinical Evidence Synthesis with Large Language Modelscited by 15
  17. 2025From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretrainingcited by 15
  18. 2023T3D: Advancing 3D Medical Vision-Language Pre-training by Learning Multi-View Visual Consistencycited by 14
  19. 2024On the Vulnerability of Applying Retrieval-Augmented Generation within Knowledge-Intensive Application Domainscited by 14
  20. 2025Towards Omni-RAG: Comprehensive Retrieval-Augmented Generation for Large Language Models in Medical Applicationscited by 13
  21. 2025MMRAG: Multi-Mode Retrieval-Augmented Generation with Large Language Models for Biomedical In-Context Learningcited by 13
  22. 2024CancerLLM: A Large Language Model in Cancer Domaincited by 11
  23. 2024RAMIE: Retrieval-Augmented Multi-task Information Extraction with Large Language Models on Dietary Supplementscited by 11
  24. 2025A foundation model for human-AI collaboration in medical literature miningcited by 11
  25. 2025RARE: Retrieval-Augmented Reasoning Modelingcited by 11
  26. 2025MedCite: Can Language Models Generate Verifiable Text for Medicine?cited by 11
  27. 2024MedCodER: A Generative AI Assistant for Medical Codingcited by 10
  28. 2025End-to-End Agentic RAG System Training for Traceable Diagnostic Reasoningcited by 10
  29. 2025Experience Retrieval-Augmentation with Electronic Health Records Enables Accurate Discharge QAcited by 9
  30. 2025Medical large language models are easily distractedcited by 9
  31. 2026LiveMedBench: A Contamination-Free Medical Benchmark for LLMs with Automated Rubric Evaluationcited by 9
  32. 2024GeneAgent: Self-verification Language Agent for Gene Set Knowledge Discovery using Domain Databasescited by 8
  33. 2024A Perspective for Adapting Generalist AI to Specialized Medical AI Applications and Their Challengescited by 8
  34. 2025Lunguage: A Benchmark for Structured and Sequential Chest X-ray Interpretationcited by 8
  35. 2024Time Matters: Examine Temporal Effects on Biomedical Language Modelscited by 7
  36. 2024Uncovering Knowledge Gaps in Radiology Report Generation Models through Knowledge Graphscited by 7
  37. 2024IryoNLP at MEDIQA-CORR 2024: Tackling the Medical Error Detection & Correction Task On the Shoulders of Medical Agentscited by 6
  38. 2024Bailicai: A Domain-Optimized Retrieval-Augmented Generation Framework for Medical Applicationscited by 6

50 most cited of 182 All 182 (JSON)