mlsanityData to 7 Oct 2026

M-FLAG: Medical Vision-Language Pre-training with Frozen Language Models and Latent Space Geometry Optimization

arXiv 2307.08347 ↗First posted 17 Jul 2023cs.CV cs.AI cs.LGcited by 43

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Medical vision-language models enable co-learning and integrating features from medical imaging and clinical text. However, these models are not easy to train and the latent representation space can be complex. Here we propose a novel way for pre-training and regularising medical vision-language models. The proposed method, named Medical vision-language pre-training with Frozen language models and Latent spAce Geometry optimization (M-FLAG), leverages a frozen language model for training stability and efficiency and introduces a novel orthogonality loss to harmonize the latent space geometry. We demonstrate the potential of the pre-trained model on three downstream tasks: medical image classification, segmentation, and object detection. Extensive experiments across five public datasets demonstrate that M-FLAG significantly outperforms existing medical vision-language pre-training approaches and reduces the number of parameters by 78%. Notably, M-FLAG achieves outstanding performance on the segmentation task while using only 1% of the RSNA dataset, even outperforming ImageNet pre-trained models that have been fine-tuned using 100% of the data.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 34 references that are in the map. First posted on arXiv 17 Jul 2023.

Builds on

26 of 34 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Generative Text-Guided 3D Vision-Language Pretraining for Unified Medical Image Segmentation7 Jun
  2. 2023Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias31 May

Other references · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2018YOLOv3: An Incremental Improvementcited by 2,652
  8. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  9. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  10. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  11. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  12. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
12 more
  1. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
  2. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  3. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  4. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  5. 2017Toward Multimodal Image-to-Image Translationcited by 295
  6. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  7. 2020CheXclusion: Fairness gaps in deep chest X-ray classifierscited by 153
  8. 2020A review: Deep learning for medical image segmentation using multi-modality fusioncited by 84
  9. 2021How to Train BERT with an Academic Budgetcited by 80
  10. 2023Frozen Language Model Helps ECG Zero-Shot Learningcited by 62
  11. 2022Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learningcited by 40
  12. 2022Latent Topology Induction for Understanding Contextualized Representationscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 21; 2025: 9; 2026: 3.

43 papers.

Sort:

By area:

  1. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  2. 2024Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancementcited by 66
  3. 2023CLIP in Medical Imaging: A Surveycited by 40
  4. 2023Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learningcited by 34
  5. 2023Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learningcited by 34
  6. 2024CT-GLIP: 3D Grounded Language-Image Pretraining with CT Scans and Radiology Reports for Full-Body Scenarioscited by 29
  7. 2023ETP: Learning Transferable ECG Representations via ECG-Text Pre-trainingcited by 28
  8. 2024BIMCV-R: A Landmark Dataset for 3D CT Text-Image Retrievalcited by 28
  9. 2023Medical Vision Language Pretraining: A surveycited by 24
  10. 2023IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-trainingcited by 23
  11. 2024MLIP: Enhancing Medical Visual Representation with Divergence Encoder and Knowledge-guided Contrastive Learningcited by 23
  12. 2023Learning Multiscale Consistency for Self-supervised Electron Microscopy Instance Segmentationcited by 16
31 more
  1. 2023Utilizing Synthetic Data for Medical Vision-Language Pre-training: Bypassing the Need for Real Imagescited by 16
  2. 2023G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-trainingcited by 12
  3. 2024DeViDe: Faceted medical knowledge for improved medical vision-language pre-trainingcited by 10
  4. 2026Efficient Chest X-ray Representation Learning via Semantic-Partitioned Contrastive Learningcited by 10
  5. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Reviewcited by 9
  6. 2024RadCLIP: Enhancing Radiologic Image Analysis through Contrastive Language-Image Pre-trainingcited by 8
  7. 2024Large Language Model-informed ECG Dual Attention Network for Heart Failure Risk Predictioncited by 8
  8. 2024MedUniSeg: 2D and 3D Medical Image Segmentation via a Prompt-driven Universal Modelcited by 7
  9. 2024Freeze the backbones: A Parameter-Efficient Contrastive Approach to Robust Medical Vision-Language Pre-trainingcited by 6
  10. 2024EEG-Language Pretraining for Highly Label-Efficient Clinical Phenotypingcited by 6
  11. 2024Insight: A Multi-Modal Diagnostic Pipeline using LLMs for Ocular Surface Disease Diagnosiscited by 6
  12. 2024Can Medical Vision-Language Pre-training Succeed with Purely Synthetic Data?cited by 6
  13. 2024BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rayscited by 6
  14. 2024MulModSeg: Enhancing Unpaired Multi-Modal Medical Image Segmentation with Modality-Conditioned Text Embedding and Alternating Trainingcited by 6
  15. 2025SuPreME: A Supervised Pre-training Framework for Multimodal ECG Representation Learningcited by 6
  16. 2024Enhancing the vision-language foundation model with key semantic knowledge-emphasized report refinementcited by 5
  17. 2023Breast Cancer Classification with Enhanced Interpretability: DALAResNet50 and DT Grad-CAMcited by 4
  18. 2024Enhancing Representation in Medical Vision-Language Foundation Models via Multi-Scale Information Extraction Techniquescited by 3
  19. 2025Black-Box Membership Inference Attack for LVLMs via Prior Knowledge-Calibrated Memory Probingcited by 3
  20. 2024Grounded Knowledge-Enhanced Medical Vision-Language Pre-training for Chest X-Raycited by 2
  21. 2024How Does Diverse Interpretability of Textual Prompts Impact Medical Vision-Language Zero-Shot Tasks?cited by 2
  22. 2025Recent Advances in Medical Image Classificationcited by 2
  23. 2025The Missing Piece: A Case for Pre-Training in 3D Medical Object Detectioncited by 2
  24. 2024MeDSLIP: Medical Dual-Stream Language-Image Pre-training with Pathology-Anatomy Semantic Alignmentcited by 1
  25. 2024Pixels to Prose: Understanding the art of Image Captioningcited by 1
  26. 2025Improving Medical Visual Representation Learning with Pathological-level Cross-Modal Alignment and Correlation Explorationcited by 1
  27. 2025FaNe: Towards Fine-Grained Cross-Modal Contrast with False-Negative Reduction and Text-Conditioned Sparse Attentioncited by 1
  28. 2026CAP: Towards PPG Universal Representation Learning with Patient-level Supervisioncited by 1
  29. 2025A Language-Signal-Vision Multimodal Framework for Multitask Cardiac Analysiscited by 0
  30. 2025TCSA-UDA: Text-Driven Cross-Semantic Alignment for Unsupervised Domain Adaptation in Medical Image Segmentationcited by 0
  31. 2026Representation Learning with Semantic-aware Instance and Sparse Token Alignmentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Reality Check of Vision-Language Pre-training in Radiology: Have We Progressed Using Text?
  2. 2025Comprehensive language-image pre-training for 3D medical image understanding
  3. 2023Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Models
  4. 2024Medical Vision-Language Pre-Training for Brain Abnormalities
  5. 2025VELVET-Med: Vision and Efficient Language Pre-training for Volumetric Imaging Tasks in Medicine
  6. 2024E3D-GPT: Enhanced 3D Visual Foundation for Medical Vision-Language Model
  7. 2025Distribution-Based Masked Medical Vision-Language Model Using Structured Reports
  8. 2026MedKCO: Medical Vision-Language Pretraining via Knowledge-Driven Cognitive Orchestration