mlsanityData to 7 Oct 2026

CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection

arXiv 2301.00785 ↗First posted 2 Jan 2023eess.IV cs.CV cs.LGcited by 224

Not on the map. Why?

Cited by 30 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

An increasing number of public datasets have shown a marked impact on automated organ segmentation and tumor detection. However, due to the small size and partially labeled problem of each dataset, as well as a limited investigation of diverse types of tumors, the resulting models are often limited to segmenting specific organs/tumors and ignore the semantics of anatomical structures, nor can they be extended to novel domains. To address these issues, we propose the CLIP-Driven Universal Model, which incorporates text embedding learned from Contrastive Language-Image Pre-training (CLIP) to segmentation models. This CLIP-based label encoding captures anatomical relationships, enabling the model to learn a structured feature embedding and segment 25 organs and 6 types of tumors. The proposed model is developed from an assembly of 14 datasets, using a total of 3,410 CT scans for training and then evaluated on 6,162 external CT scans from 3 additional datasets. We rank first on the Medical Segmentation Decathlon (MSD) public leaderboard and achieve state-of-the-art results on Beyond The Cranial Vault (BTCV). Additionally, the Universal Model is computationally more efficient (6x faster) compared with dataset-specific models, generalized better to CT scans from varying sites, and shows stronger transfer learning performance on novel tasks.

Key facts. Cited by 224 papers in the map (data through 7 Oct 2026). Builds on 62 of its 120 references that are in the map. First posted on arXiv 2 Jan 2023.

Builds on

62 of 120 references.

References · 62

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  5. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  7. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,552
  8. 2018UNet++: A Nested U-Net Architecture for Medical Image Segmentationcited by 1,518
  9. 2018Attention U-Net: Learning Where to Look for the Pancreascited by 1,420
  10. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  11. 2021UNETR: Transformers for 3D Medical Image Segmentationcited by 954
  12. 2022Language-driven Semantic Segmentationcited by 711
38 more
  1. 2021The Medical Segmentation Decathloncited by 620
  2. 2019UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentationcited by 583
  3. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  4. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  5. 2021Image Segmentation Using Text and Image Promptscited by 509
  6. 2019The Liver Tumor Segmentation Benchmark (LiTS)cited by 504
  7. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 418
  8. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  9. 2022AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentationcited by 347
  10. 2020Conditional Convolutions for Instance Segmentationcited by 314
  11. 2019Med3D: Transfer Learning for 3D Medical Image Analysiscited by 282
  12. 2021TransBTS: Multimodal Brain Tumor Segmentation Using Transformercited by 265
  13. 2019Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentationcited by 258
  14. 2021CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentationcited by 227
  15. 2020AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?cited by 222
  16. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 209
  17. 2019The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomescited by 168
  18. 2019Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysiscited by 159
  19. 2015DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentationcited by 158
  20. 2021Recent advances and clinical applications of deep learning in medical image analysiscited by 104
  21. 2020DoDNet: Learning to segment multi-organ and tumors from multiple partially labeled datasetscited by 94
  22. 2019Prior-aware Neural Network for Partially-Supervised Multi-Organ Segmentationcited by 88
  23. 2016Entity Embeddings of Categorical Variablescited by 85
  24. 2021Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?cited by 85
  25. 2022Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domainscited by 85
  26. 2020Annotation-efficient deep learning for automatic medical image segmentationcited by 79
  27. 2021Multi-institutional Collaborations for Improving Deep Learning-based Magnetic Resonance Image Reconstruction Using Federated Learningcited by 78
  28. 2018Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapycited by 76
  29. 2021Template-free Prompt Tuning for Few-shot NERcited by 73
  30. 2020Multi-organ Segmentation over Partially Labeled Datasets with Multi-scale Feature Abstractioncited by 70
  31. 2021Transferable Visual Words: Exploiting the Semantics of Anatomical Patterns for Self-supervised Learningcited by 68
  32. 2022Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Studycited by 66
  33. 2021UniMiSS: Universal Medical Self-Supervised Learning via Breaking Dimensionality Barriercited by 55
  34. 2019C2FNAS: Coarse-to-Fine Neural Architecture Search for 3D Medical Image Segmentationcited by 49
  35. 2021DiNTS: Differentiable Neural Network Topology Search for 3D Medical Image Segmentationcited by 47
  36. 2020Learning from Multiple Datasets with Heterogeneous and Partial Labels for Universal Lesion Detection in CTcited by 42
  37. 2022Per-Clip Video Object Segmentationcited by 40
  38. 2019Partial Order Pruning: for Best Speed/Accuracy Trade-off in Neural Architecture Searchcited by 39

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 36; 2024: 83; 2025: 67; 2026: 38.

224 papers.

Sort:

By area:

  1. 2023AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detectioncited by 270
  2. 2023SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Imagescited by 101
  3. 2023SegVol: Universal and Interactive Volumetric Medical Image Segmentationcited by 86
  4. 2024MAISI: Medical AI for Synthetic Imagingcited by 86
  5. 2023A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervisioncited by 78
  6. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  7. 2023Foundational Models in Medical Imaging: A Comprehensive Survey and Future Visioncited by 74
  8. 2025AA-CLIP: Enhancing Zero-shot Anomaly Detection via Anomaly-Aware CLIPcited by 72
  9. 2024FairCLIP: Harnessing Fairness in Vision-Language Learningcited by 69
  10. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Modelscited by 66
  11. 2024Towards Generalizable Tumor Synthesiscited by 65
  12. 2024AbdomenAtlas: A Large-Scale, Detailed-Annotated, & Multi-Center Dataset for Efficient Transfer Learning and Open Algorithmic Benchmarkingcited by 64
38 more
  1. 2024VISTA3D: A Unified Segmentation Foundation Model For 3D Medical Imagingcited by 62
  2. 2023Label-Free Liver Tumor Segmentationcited by 56
  3. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  4. 2025Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Modelscited by 54
  5. 2023Generative Text-Guided 3D Vision-Language Pretraining for Unified Medical Image Segmentationcited by 47
  6. 2023UniSeg: A Prompt-driven Universal Segmentation Model as well as A Strong Representation Learnercited by 46
  7. 2023MultiTalent: A Multi-Dataset Approach to Medical Image Segmentationcited by 45
  8. 2023Continual Learning for Abdominal Multi-Organ and Tumor Segmentationcited by 43
  9. 2023CLIP in Medical Imaging: A Surveycited by 40
  10. 2024Mammo-CLIP: A Vision Language Foundation Model to Enhance Data Efficiency and Robustness in Mammographycited by 35
  11. 2023Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learningcited by 34
  12. 2024Large-Scale 3D Medical Image Pre-training with Geometric Context Priorscited by 34
  13. 2025RadGPT: Constructing 3D Image-Text Tumor Datasetscited by 34
  14. 2024WATT: Weight Average Test-Time Adaptation of CLIPcited by 33
  15. 2023Training Like a Medical Resident: Context-Prior Learning Toward Universal Medical Image Segmentationcited by 32
  16. 2024Universal and Extensible Language-Vision Models for Organ Segmentation and Tumor Detection from Abdominal Computed Tomographycited by 31
  17. 2024Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?cited by 31
  18. 2025Medical World Model: Generative Simulation of Tumor Evolution for Treatment Planningcited by 31
  19. 2023SegRap2023: A Benchmark of Organs-at-Risk and Gross Tumor Volume Segmentation for Radiotherapy Planning of Nasopharyngeal Carcinomacited by 28
  20. 2024Foundation Models for Biomedical Image Segmentation: A Surveycited by 28
  21. 2024Robust Calibration of Large Vision-Language Adapterscited by 28
  22. 2023Text Promptable Surgical Instrument Segmentation with Vision-Language Modelscited by 27
  23. 2024Data-Centric Foundation Models in Computational Healthcare: A Surveycited by 27
  24. 2024Learning General-Purpose Biomedical Volume Representations using Randomized Synthesiscited by 27
  25. 2023Deep Interactive Segmentation of Medical Images: A Systematic Review and Taxonomycited by 26
  26. 2023MIS-FM: 3D Medical Image Segmentation using Foundation Models Pretrained on a Large-Scale Unannotated Datasetcited by 25
  27. 2024Test-Time Adaptation with SaLIP: A Cascade of SAM and CLIP for Zero shot Medical Image Segmentationcited by 25
  28. 2025MedSAM3: Delving into Segment Anything with Medical Conceptscited by 25
  29. 2024MM-Retinal: Knowledge-Enhanced Foundational Pretraining with Fundus Image-Text Expertisecited by 24
  30. 2024Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Modelscited by 23
  31. 2024From Pixel to Cancer: Cellular Automata in Computed Tomographycited by 20
  32. 2024Automatic Organ and Pan-cancer Segmentation in Abdomen CT: the FLARE 2023 Challengecited by 20
  33. 2025How Well Do Supervised 3D Models Transfer to Medical Imaging Tasks?cited by 19
  34. 2023Spatially Covariant Image Registration with Text Promptscited by 18
  35. 2024Spider: A Unified Framework for Context-dependent Concept Segmentationcited by 18
  36. 2023Towards Foundation Models and Few-Shot Parameter-Efficient Fine-Tuning for Volumetric Organ Segmentationcited by 17
  37. 2023Early Detection and Localization of Pancreatic Cancer by Label-Free Tumor Synthesiscited by 17
  38. 2024MiM: Mask in Mask Self-Supervised Pre-Training for 3D Medical Image Analysiscited by 17

50 most cited of 224 All 224 (JSON)