mlsanityData to 14 Oct 2026

Generative Text-Guided 3D Vision-Language Pretraining for Unified Medical Image Segmentation

arXiv 2306.04811 ↗First posted 7 Jun 2023cs.CV cs.AIcited by 47

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Vision-Language Pretraining (VLP) has demonstrated remarkable capabilities in learning visual representations from textual descriptions of images without annotations. Yet, effective VLP demands large-scale image-text pairs, a resource that suffers scarcity in the medical domain. Moreover, conventional VLP is limited to 2D images while medical images encompass diverse modalities, often in 3D, making the learning process more challenging. To address these challenges, we present Generative Text-Guided 3D Vision-Language Pretraining for Unified Medical Image Segmentation (GTGM), a framework that extends of VLP to 3D medical images without relying on paired textual descriptions. Specifically, GTGM utilizes large language models (LLM) to generate medical-style text from 3D medical images. This synthetic text is then used to supervise 3D visual representation learning. Furthermore, a negative-free contrastive learning objective strategy is introduced to cultivate consistent visual representations between augmented 3D medical image patches, which effectively mitigates the biases associated with strict positive-negative sample pairings. We evaluate GTGM on three imaging modalities - Computed Tomography (CT), Magnetic Resonance Imaging (MRI), and electron microscopy (EM) over 13 datasets. GTGM's superior performance across various medical image segmentation tasks underscores its effectiveness and versatility, by enabling VLP extension into 3D medical imagery while bypassing the need for paired text.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 53 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

36 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias31 May

Other references · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  11. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  12. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
23 more
  1. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  2. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  3. 2016Colorful Image Colorizationcited by 1,114
  4. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  5. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  6. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  7. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  8. 2021The Medical Segmentation Decathloncited by 628
  9. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 422
  10. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 339
  11. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  12. 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
  13. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  14. 2023UniverSeg: Universal Medical Image Segmentationcited by 91
  15. 2023Advancing Radiograph Representation Learning with Masked Record Modelingcited by 75
  16. 2023Machine learning with data assimilation and uncertainty quantification for dynamical systems: a reviewcited by 75
  17. 2020MedICaT: A Dataset of Medical Images, Captions, and Textual Referencescited by 71
  18. 2023Frozen Language Model Helps ECG Zero-Shot Learningcited by 62
  19. 2023UniSeg: A Prompt-driven Universal Segmentation Model as well as A Strong Representation Learnercited by 47
  20. 2023PCRLv2: A Unified Visual Information Preservation Framework for Self-supervised Pre-training in Medical Image Analysiscited by 40
  21. 2022Stare at What You See: Masked Image Modeling without Reconstructioncited by 30
  22. 2013Machine learning of hierarchical clustering to segment 2D and 3D imagescited by 11
  23. 2022Spatio-Temporal Contrastive Learning Enhanced GNNs for Session-based Recommendationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 30; 2025: 6; 2026: 1.

47 papers.

Sort:

By area:

  1. 2024Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancementcited by 67
  2. 2024Endora: Video Generation Models as Endoscopy Simulatorscited by 55
  3. 2023M-FLAG: Medical Vision-Language Pre-training with Frozen Language Models and Latent Space Geometry Optimizationcited by 43
  4. 2024Accelerating Parallel Sampling of Diffusion Modelscited by 30
  5. 2024BIMCV-R: A Landmark Dataset for 3D CT Text-Image Retrievalcited by 30
  6. 2023ETP: Learning Transferable ECG Representations via ECG-Text Pre-trainingcited by 29
  7. 2025Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Toolscited by 26
  8. 2023Medical Vision Language Pretraining: A surveycited by 24
  9. 2024CU-Net: a U-Net architecture for efficient brain-tumor segmentation on BraTS 2019 datasetcited by 24
  10. 2023IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-trainingcited by 23
  11. 2023Self-Supervised Neuron Segmentation with Multi-Agent Reinforcement Learningcited by 19
  12. 2025Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Drivingcited by 19
35 more
  1. 2024Research on Adverse Drug Reaction Prediction Model Combining Knowledge Graph Embedding and Deep Learningcited by 17
  2. 2023Learning Multiscale Consistency for Self-supervised Electron Microscopy Instance Segmentationcited by 16
  3. 2023Utilizing Synthetic Data for Medical Vision-Language Pre-training: Bypassing the Need for Real Imagescited by 16
  4. 2024Rumor Detection with a novel graph neural network approachcited by 15
  5. 2024Mask Factory: Towards High-quality Synthetic Data Generation for Dichotomous Image Segmentationcited by 15
  6. 2023T3D: Advancing 3D Medical Vision-Language Pre-training by Learning Multi-View Visual Consistencycited by 14
  7. 2024Image Captioning in news report scenariocited by 12
  8. 2024Application of Multimodal Fusion Deep Learning Model in Disease Recognitioncited by 12
  9. 2024A comparative study of generative adversarial networks for image recognition algorithms based on deep learning and traditional methodscited by 12
  10. 2024Transformer-Based Classification Outcome Prediction for Multimodal Stroke Treatmentcited by 11
  11. 2024Research on Image Recognition Technology Based on Multimodal Deep Learningcited by 11
  12. 2024Enhancing Medical Imaging with GANs Synthesizing Realistic Images from Limited Datacited by 11
  13. 2024Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Groundingcited by 10
  14. 2024Applying Conditional Generative Adversarial Networks for Imaging Diagnosiscited by 9
  15. 2024MG-3D: Multi-Grained Knowledge-Enhanced 3D Medical Vision-Language Pre-trainingcited by 8
  16. 2024EEG-Language Pretraining for Highly Label-Efficient Clinical Phenotypingcited by 6
  17. 2024VoxelPrompt: A Vision Agent for End-to-End Medical Image Analysiscited by 6
  18. 2024MulModSeg: Enhancing Unpaired Multi-Modal Medical Image Segmentation with Modality-Conditioned Text Embedding and Alternating Trainingcited by 6
  19. 2023Unified Medical Image Pre-training in Language-Guided Common Semantic Spacecited by 5
  20. 2024Med3DInsight: Enhancing 3D Medical Image Understanding with 2D Multi-Modal Large Language Modelscited by 5
  21. 2024Investigation of Customized Medical Decision Algorithms Utilizing Graph Neural Networkscited by 5
  22. 2024CAR-MFL: Cross-Modal Augmentation by Retrieval for Multimodal Federated Learning with Missing Modalitiescited by 5
  23. 2023Breast Cancer Classification with Enhanced Interpretability: DALAResNet50 and DT Grad-CAMcited by 4
  24. 2024Analyzing Diversity in Healthcare LLM Research: A Scientometric Perspectivecited by 4
  25. 2024EHRmonize: A Framework for Medical Concept Abstraction from Electronic Health Records using Large Language Modelscited by 4
  26. 2024Leveraging Task-Specific Knowledge from LLM for Semi-Supervised 3D Medical Image Segmentationcited by 4
  27. 2025DVP-MVS++: Synergize Depth-Normal-Edge and Harmonized Visibility Prior for Multi-View Stereocited by 4
  28. 2024UniCompress: Enhancing Multi-Data Medical Image Compression with Knowledge Distillationcited by 3
  29. 2024End-To-End Training and Testing Gamification Framework to Learn Human Highway Drivingcited by 2
  30. 2024How Does Diverse Interpretability of Textual Prompts Impact Medical Vision-Language Zero-Shot Tasks?cited by 2
  31. 2024E-commerce Webpage Recommendation Scheme Base on Semantic Mining and Neural Networkscited by 1
  32. 2025BiPVL-Seg: Bidirectional Progressive Vision-Language Fusion with Global-Local Alignment for Medical Image Segmentationcited by 1
  33. 2025Privacy-Shielded Image Compression: Defending Against Exploitation from Vision-Language Pretrained Modelscited by 0
  34. 2025TCSA-UDA: Text-Driven Cross-Semantic Alignment for Unsupervised Domain Adaptation in Medical Image Segmentationcited by 0
  35. 2026ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalitiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Comprehensive language-image pre-training for 3D medical image understanding
  2. 2023Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Models
  3. 2026Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography
  4. 2026MedPlex: Deep Vision-Language Co-Adaptation for Clinically Grounded Medical Segmentation
  5. 2026Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT
  6. 2025A Reality Check of Vision-Language Pre-training in Radiology: Have We Progressed Using Text?
  7. 2026MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation
  8. 2024E3D-GPT: Enhanced 3D Visual Foundation for Medical Vision-Language Model