mlsanityData to 7 Oct 2026

Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing

arXiv 2301.04558 ↗First posted 11 Jan 2023cs.CV cs.CLcited by 210

Not on the map. Why?

Cited by 57 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Self-supervised learning in vision-language processing exploits semantic alignment between imaging and text modalities. Prior work in biomedical VLP has mostly relied on the alignment of single image and report pairs even though clinical notes commonly refer to prior images. This does not only introduce poor alignment between the modalities but also a missed opportunity to exploit rich self-supervision through existing temporal content in the data. In this work, we explicitly account for prior images and reports when available during both training and fine-tuning. Our approach, named BioViL-T, uses a CNN-Transformer hybrid multi-image encoder trained jointly with a text model. It is designed to be versatile to arising challenges such as pose variations and missing input images across time. The resulting model excels on downstream tasks both in single- and multi-image setups, achieving state-of-the-art performance on (I) progression classification, (II) phrase grounding, and (III) report generation, whilst offering consistent improvements on disease classification and sentence-similarity tasks. We release a novel multi-modal temporal benchmark dataset, MS-CXR-T, to quantify the quality of vision-language representations in terms of temporal semantics. Our experimental results show the advantages of incorporating prior images and reports to make most use of the data.

Key facts. Cited by 210 papers in the map (data through 7 Oct 2026). Builds on 52 of its 88 references that are in the map. First posted on arXiv 11 Jan 2023.

Builds on

52 of 88 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2016Layer Normalizationcited by 5,579
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  4. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  5. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  6. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  7. 2020Quantifying Attention Flow in Transformerscited by 759
  8. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  9. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  10. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  11. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  12. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  13. 2020Generating Radiology Reports via Memory-driven Transformercited by 363
  14. 2020Debiased Contrastive Learningcited by 354
  15. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  16. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  17. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  18. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
  19. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
  20. 2018Fully Convolutional Siamese Networks for Change Detectioncited by 305
  21. 2022How Do Vision Transformers Work?cited by 305
  22. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  23. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  24. 2020Self-supervised Co-training for Video Representation Learningcited by 232
  25. 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 206
  26. 2021Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIPcited by 151
  27. 2020Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generationcited by 139
  28. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108
  29. 2021Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Trainingcited by 96
  30. 2021MedAug: Contrastive learning leveraging patient metadata improves representations for chest X-ray interpretationcited by 52
  31. 2022Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priorscited by 48
  32. 2022Robust Contrastive Learning against Noisy Viewscited by 45
  33. 2021COVID-19 Prognosis via Self-Supervised Representation Learning and Multi-Image Predictioncited by 36
  34. 2021Long Short View Feature Decomposition via Contrastive Video Representation Learningcited by 21
  35. 2022CheXRelNet: An Anatomy-Aware Model for Tracking Longitudinal Relationships between Chest X-Rayscited by 19
  36. 2021Data Efficient Masked Language Modeling for Vision and Languagecited by 18
  37. 2022Time Is MattEr: Temporal Self-supervision for Video Transformerscited by 11
  38. 2018Longitudinal detection of radiological abnormalities with time-modulated LSTMcited by 9

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 25; 2024: 48; 2025: 67; 2026: 70.

210 papers.

Sort:

By area:

  1. 2023Towards Generalist Biomedical AIcited by 286
  2. 2024MAIRA-2: Grounded Radiology Report Generationcited by 140
  3. 2024Exploring scalable medical image encoders beyond text supervisioncited by 129
  4. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 121
  5. 2024CheXpert Plus: Augmenting a Large Chest X-ray Dataset with Text Radiology Reports, Patient Demographics and Additional Image Formatscited by 100
  6. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  7. 2023Foundational Models in Medical Imaging: A Comprehensive Survey and Future Visioncited by 74
  8. 2023MAIRA-1: A specialised large multimodal model for radiology report generationcited by 72
  9. 2024RaTEScore: A Metric for Radiology Report Generationcited by 65
  10. 2023MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Textscited by 51
  11. 2024Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluationcited by 51
  12. 2024Automated Radiology Report Generation: A Review of Recent Advancescited by 43
38 more
  1. 2026MedGemma 1.5 Technical Reportcited by 43
  2. 2023CLIP in Medical Imaging: A Surveycited by 40
  3. 2024CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classificationcited by 40
  4. 2025Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generationcited by 39
  5. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  6. 2024BrainSegFounder: Towards 3D Foundation Models for Neuroimage Segmentationcited by 36
  7. 2023Towards Unifying Medical Vision-and-Language Pre-training via Soft Promptscited by 35
  8. 2023Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learningcited by 34
  9. 2024HERGen: Elevating Radiology Report Generation with Longitudinal Datacited by 32
  10. 2023Exploring the Boundaries of GPT-4 in Radiologycited by 31
  11. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  12. 2024Libra: Leveraging Temporal Images for Biomedical Radiology Analysiscited by 29
  13. 2023RadEdit: stress-testing biomedical vision models via diffusion image editingcited by 28
  14. 2024MiniGPT-Med: Large Language Model as a General Interface for Radiology Diagnosiscited by 28
  15. 2024Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Frameworkcited by 27
  16. 2023Longitudinal Data and a Semantic Similarity Reward for Chest X-Ray Report Generationcited by 24
  17. 2023Medical Vision Language Pretraining: A surveycited by 24
  18. 2023Controllable Chest X-Ray Report Generation from Longitudinal Representationscited by 23
  19. 2024An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generationcited by 23
  20. 2024Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalitiescited by 23
  21. 2023RECAP: Towards Precise Radiology Report Generation via Dynamic Disease Progression Reasoningcited by 22
  22. 2025RadEval: A framework for radiology text evaluationcited by 21
  23. 2024Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Surveycited by 20
  24. 2025Comprehensive language-image pre-training for 3D medical image understandingcited by 20
  25. 2024ChEX: Interactive Localization and Region Description in Chest X-rayscited by 18
  26. 2024EchoPrime: A Multi-Video View-Informed Vision-Language Model for Comprehensive Echocardiography Interpretationcited by 17
  27. 2024Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directionscited by 17
  28. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  29. 2024EchoFM: Foundation Model for Generalizable Echocardiogram Analysiscited by 14
  30. 2024Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-trainingcited by 13
  31. 2025Towards Scalable Language-Image Pre-training for 3D Medical Imagingcited by 13
  32. 2023Longformer: Longitudinal Transformer for Alzheimer's Disease Classification with Structural MRIscited by 12
  33. 2023Consensus, dissensus and synergy between clinicians and specialist foundation models in radiology report generationcited by 12
  34. 2025LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understandingcited by 12
  35. 2025CoCa-CXR: Contrastive Captioners Learn Strong Temporal Structures for Chest X-Ray Vision-Language Understandingcited by 11
  36. 2025MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representationscited by 11
  37. 2025RADAR: Enhancing Radiology Report Generation with Supplementary Knowledge Injectioncited by 11
  38. 2025Bringing CLIP to the Clinic: Dynamic Soft Labels and Negation-Aware Learning for Medical Analysiscited by 10

50 most cited of 210 All 210 (JSON)