mlsanityData to 7 Oct 2026

MedICaT: A Dataset of Medical Images, Captions, and Textual References

arXiv 2010.06000 ↗First posted 12 Oct 2020cs.CV cs.CLcited by 71

Not on the map. Why?

Cited by 30 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Understanding the relationship between figures and text is key to scientific document understanding. Medical figures in particular are quite complex, often consisting of several subfigures (75% of figures in our dataset), with detailed text describing their content. Previous work studying figures in scientific papers focused on classifying figure content rather than understanding how images relate to the text. To address challenges in figure retrieval and figure-to-text alignment, we introduce MedICaT, a dataset of medical images in context. MedICaT consists of 217K images from 131K open access biomedical papers, and includes captions, inline references for 74% of figures, and manually annotated subfigures and subcaptions for a subset of figures. Using MedICaT, we introduce the task of subfigure to subcaption alignment in compound figures and demonstrate the utility of inline references in image-text matching. Our data and code can be accessed at https://github.com/allenai/medicat.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 14; 2024: 29; 2025: 17; 2026: 8.

71 papers.

Sort:

By area:

  1. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  2. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 217
  3. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  4. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  5. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 121
  6. 2022Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Trainingcited by 104
  7. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  8. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  9. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  10. 2024FairCLIP: Harnessing Fairness in Vision-Language Learningcited by 69
  11. 2023PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathologycited by 57
  12. 2024ROCOv2: Radiology Objects in COntext Version 2, an Updated Multimodal Image Datasetcited by 54
38 more
  1. 2022Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledgecited by 51
  2. 2023Generative Text-Guided 3D Vision-Language Pretraining for Unified Medical Image Segmentationcited by 47
  3. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  4. 2023CLIP in Medical Imaging: A Surveycited by 40
  5. 2023Towards Unifying Medical Vision-and-Language Pre-training via Soft Promptscited by 35
  6. 2025BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literaturecited by 34
  7. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  8. 2023Masked Vision and Language Pre-training with Unimodal and Multimodal Contrastive Losses for Medical Visual Question Answeringcited by 29
  9. 2024Masked Image Modeling: A Surveycited by 28
  10. 2023Medical Vision Language Pretraining: A surveycited by 24
  11. 2024GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AIcited by 23
  12. 2024PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imagingcited by 21
  13. 2024MedM2G: Unifying Medical Multi-Modal Generation via Cross-Guided Diffusion with Visual Invariantcited by 21
  14. 2024Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Surveycited by 20
  15. 2024Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQAcited by 19
  16. 2025A Survey on Efficient Vision-Language Modelscited by 19
  17. 2024Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directionscited by 17
  18. 2024A Spectrum Evaluation Benchmark for Medical Multi-Modal Large Language Modelscited by 16
  19. 2024Large Language Model Benchmarks in Medical Taskscited by 16
  20. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  21. 2024Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via Cross-modal Alignmentcited by 15
  22. 2024MISS: A Generative Pretraining and Finetuning Approach for Med-VQAcited by 14
  23. 2024LaPA: Latent Prompt Assist Model For Medical Visual Question Answeringcited by 12
  24. 2024Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcarecited by 12
  25. 2025UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysiscited by 12
  26. 2024Efficiency in Focus: LayerNorm as a Catalyst for Fine-tuning Medical Visual Language Pre-trained Modelscited by 11
  27. 2025Fleming-VL: Towards Universal Medical Visual Reasoning with Multimodal LLMscited by 10
  28. 2025Open-PMC-18M: A High-Fidelity Large Scale Medical Dataset for Multimodal Representation Learningcited by 9
  29. 2024Medical MLLM is Vulnerable: Cross-Modality Jailbreak and Mismatched Attacks on Medical Multimodal Large Language Modelscited by 8
  30. 2024Dragonfly: Multi-Resolution Zoom-In Encoding Enhances Vision-Language Modelscited by 8
  31. 2024LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasoundcited by 8
  32. 2025Advancing Medical Representation Learning Through High-Quality Datacited by 8
  33. 2025Generative Artificial Intelligence in Medical Imaging: Foundations, Progress, and Clinical Translationcited by 8
  34. 2023Sam-Guided Enhanced Fine-Grained Encoding with Mixed Semantic Learning for Medical Image Captioningcited by 7
  35. 2025GAIA: A Global, Multi-modal, Multi-scale Vision-Language Dataset for Remote Sensing Image Analysiscited by 7
  36. 2024MedPix 2.0: A Comprehensive Multimodal Biomedical Data set for Advanced AI Applications with Retrieval Augmented Generation and Knowledge Graphscited by 6
  37. 2024Visual Prompt Engineering for Vision Language Models in Radiologycited by 6
  38. 2023Multi-modal Pre-training for Medical Vision-language Understanding and Generation: An Empirical Study with A New Benchmarkcited by 5

50 most cited of 71 All 71 (JSON)