mlsanityData to 14 Oct 2026

MOSMOS: Multi-organ segmentation facilitated by medical report supervision

arXiv 2409.02418 ↗First posted 4 Sep 2024cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Owing to a large amount of multi-modal data in modern medical systems, such as medical images and reports, Medical Vision-Language Pre-training (Med-VLP) has demonstrated incredible achievements in coarse-grained downstream tasks (i.e., medical classification, retrieval, and visual question answering). However, the problem of transferring knowledge learned from Med-VLP to fine-grained multi-organ segmentation tasks has barely been investigated. Multi-organ segmentation is challenging mainly due to the lack of large-scale fully annotated datasets and the wide variation in the shape and size of the same organ between individuals with different diseases. In this paper, we propose a novel pre-training & fine-tuning framework for Multi-Organ Segmentation by harnessing Medical repOrt Supervision (MOSMOS). Specifically, we first introduce global contrastive learning to maximally align the medical image-report pairs in the pre-training stage. To remedy the granularity discrepancy, we further leverage multi-label recognition to implicitly learn the semantic correspondence between image pixels and organ tags. More importantly, our pre-trained models can be transferred to any segmentation model by introducing the pixel-tag attention maps. Different network settings, i.e., 2D U-Net and 3D UNETR, are utilized to validate the generalization. We have extensively evaluated our approach using different diseases and modalities on BTCV, AMOS, MMWHS, and BRATS datasets. Experimental results in various settings demonstrate the effectiveness of our framework. This framework can serve as the foundation to facilitate future research on automatic annotation tasks under the supervision of medical reports.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 65 references that are in the map. First posted on arXiv 4 Sep 2024.

Builds on

47 of 65 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  12. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
35 more
  1. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  2. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,558
  3. 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,222
  4. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  5. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  6. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  7. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  8. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 431
  9. 2019A large annotated medical image dataset for the development and evaluation of segmentation algorithmscited by 373
  10. 2022AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentationcited by 351
  11. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 339
  12. 2023Recognize Anything: A Strong Image Tagging Modelcited by 309
  13. 2019Med3D: Transfer Learning for 3D Medical Image Analysiscited by 288
  14. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  15. 20203D Self-Supervised Methods for Medical Imagingcited by 134
  16. 2021Query2Label: A Simple Transformer Way to Multi-Label Classificationcited by 128
  17. 2022LViT: Language meets Vision Transformer in Medical Image Segmentationcited by 111
  18. 2020DoDNet: Learning to segment multi-organ and tumors from multiple partially labeled datasetscited by 94
  19. 2023Tag2Text: Guiding Vision-Language Model via Image Taggingcited by 86
  20. 2021Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?cited by 85
  21. 2020Unpaired Multi-modal Segmentation via Knowledge Distillationcited by 81
  22. 2021Multi-Label Learning from Single Positive Labelscited by 81
  23. 2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
  24. 2021Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reportscited by 80
  25. 2020Multi-organ Segmentation over Partially Labeled Datasets with Multi-scale Feature Abstractioncited by 70
  26. 2023Open-Set Image Tagging with Multi-Grained Text Supervisioncited by 67
  27. 2022Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledgecited by 51
  28. 2018Abdominal multi-organ segmentation with organ-attention networks and statistical fusioncited by 46
  29. 2023Continual Learning for Abdominal Multi-Organ and Tumor Segmentationcited by 43
  30. 2024Universal and Extensible Language-Vision Models for Organ Segmentation and Tumor Detection from Abdominal Computed Tomographycited by 31
  31. 2021Multimodal Representation Learning via Maximization of Local Mutual Informationcited by 27
  32. 2022Breaking with Fixed Set Pathology Recognition through Report-Guided Contrastive Trainingcited by 26
  33. 2022Learning from partially labeled data for multi-organ and tumor segmentationcited by 18
  34. 2024Self-Supervised Learning for Medical Image Data with Anatomy-Oriented Imaging Planescited by 17
  35. 2022IDEA: Increasing Text Diversity via Online Multi-Label Recognition for Vision-Language Pre-trainingcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Leveraging Transfer Learning with Class-Specific Decoders for Laparoscopic Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021MS-KD: Multi-Organ Segmentation with Multiple Binary-Labeled Datasets
  2. 2026MedPlex: Deep Vision-Language Co-Adaptation for Clinically Grounded Medical Segmentation
  3. 2026ReportMedSAM: Guiding Segmentation Through Radiology Reports
  4. 2026SegBanana: Steering Unified Multimodal Models into Medical Segmenters
  5. 2025Anatomy-VLM: A Fine-grained Vision-Language Model for Medical Interpretation
  6. 2023Generative Text-Guided 3D Vision-Language Pretraining for Unified Medical Image Segmentation
  7. 2023Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Models
  8. 2026SAM Meets VLM: Parameter-Decoupled Full-Parameter Training for Unified Medical Reasoning and Segmentation