mlsanityData to 14 Oct 2026

MultiTalent: A Multi-Dataset Approach to Medical Image Segmentation

arXiv 2303.14444 ↗First posted 25 Mar 2023eess.IV cs.CVcited by 45

Not on the map. Why?

Abstract (arXiv)

The medical imaging community generates a wealth of datasets, many of which are openly accessible and annotated for specific diseases and tasks such as multi-organ or lesion segmentation. Current practices continue to limit model training and supervised pre-training to one or a few similar datasets, neglecting the synergistic potential of other available annotated data. We propose MultiTalent, a method that leverages multiple CT datasets with diverse and conflicting class definitions to train a single model for a comprehensive structure segmentation. Our results demonstrate improved segmentation performance compared to previous related approaches, systematically, also compared to single dataset training using state-of-the-art methods, especially for lesion segmentation and other challenging structures. We show that MultiTalent also represents a powerful foundation model that offers a superior pre-training for various segmentation tasks compared to commonly used supervised or unsupervised pre-training baselines. Our findings offer a new direction for the medical imaging community to effectively utilize the wealth of available data for improved segmentation performance. The code and model weights will be published here: [tba]

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Builds on 21 of its 32 references that are in the map. First posted on arXiv 25 Mar 2023.

Builds on

21 of 32 references.

References · 21

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2021UNETR: Transformers for 3D Medical Image Segmentationcited by 959
  3. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  4. 2021The Medical Segmentation Decathloncited by 628
  5. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 422
  6. 2022AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentationcited by 351
  7. 2019Med3D: Transfer Learning for 3D Medical Image Analysiscited by 288
  8. 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
  9. 2019The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomescited by 172
  10. 2015DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentationcited by 159
  11. 2020DoDNet: Learning to segment multi-organ and tumors from multiple partially labeled datasetscited by 94
  12. 2019Prior-aware Neural Network for Partially-Supervised Multi-Organ Segmentationcited by 88
9 more
  1. 2020Multi-organ Segmentation over Partially Labeled Datasets with Multi-scale Feature Abstractioncited by 70
  2. 2020Multi-organ Segmentation via Co-training Weight-averaged Models from Few-organ Datasetscited by 26
  3. 2022Extending nnU-Net is all you needcited by 22
  4. 2019An attempt at beating the 3D U-Netcited by 21
  5. 2021Label-set Loss Functions for Partial Supervision: Application to Fetal Brain 3D MRI Parcellationcited by 12
  6. 2021MS-KD: Multi-Organ Segmentation with Multiple Binary-Labeled Datasetscited by 8
  7. 2019Joint Learning of Brain Lesion and Anatomy Segmentation from Heterogeneous Datasetscited by 5
  8. 2021Learning from Partially Overlapping Labels: Image Segmentation under Annotation Shiftcited by 3
  9. 2023Multi-organ segmentation: a progressive exploration of learning paradigms under scarce annotationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 19; 2025: 16; 2026: 5.

45 papers.

Sort:

By area:

  1. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  2. 2025nnInteractive: Redefining 3D Promptable Segmentationcited by 41
  3. 2024Revisiting MAE pre-training for 3D medical image segmentationcited by 35
  4. 2023Training Like a Medical Resident: Context-Prior Learning Toward Universal Medical Image Segmentationcited by 33
  5. 2024Universal and Extensible Language-Vision Models for Organ Segmentation and Tumor Detection from Abdominal Computed Tomographycited by 31
  6. 2024Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?cited by 31
  7. 2023SegRap2023: A Benchmark of Organs-at-Risk and Gross Tumor Volume Segmentation for Radiotherapy Planning of Nasopharyngeal Carcinomacited by 29
  8. 2024An OpenMind for 3D medical vision self-supervised learningcited by 26
  9. 2025Primus: Enforcing Attention Usage for 3D Medical Image Segmentationcited by 24
  10. 2023Towards Foundation Models and Few-Shot Parameter-Efficient Fine-Tuning for Volumetric Organ Segmentationcited by 17
  11. 2023COSST: Multi-organ Segmentation with Partially Labeled Datasets Using Comprehensive Supervisions and Self-trainingcited by 16
  12. 2024From FDG to PSMA: A Hitchhiker's Guide to Multitracer, Multicenter Lesion Segmentation in PET/CT Imagingcited by 16
33 more
  1. 2025Show and Segment: Universal Medical Image Segmentation via In-Context Learningcited by 16
  2. 2024A Foundation Model for Brain Lesion Segmentation with Mixture of Modality Expertscited by 14
  3. 2025LesionLocator: Zero-Shot Universal Tumor Segmentation and Tracking in 3D Whole-Body Imagingcited by 12
  4. 2024Feasibility and benefits of joint learning from MRI databases with different brain diseases and modalities for segmentationcited by 8
  5. 2024Real World Federated Learning with a Knowledge Distilled Transformer for Cardiac CT Imagingcited by 8
  6. 2024MedUniSeg: 2D and 3D Medical Image Segmentation via a Prompt-driven Universal Modelcited by 7
  7. 2024Feasibility of Federated Learning from Client Databases with Different Brain Diseases and MRI Modalitiescited by 5
  8. 2024FUNAvg: Federated Uncertainty Weighted Averaging for Datasets with Diverse Labelscited by 5
  9. 2025MedNeXt-v2: Scaling 3D ConvNeXts for Large-Scale Supervised Representation Learning in Medical Image Segmentationcited by 5
  10. 2024RadioActive: 3D Radiological Interactive Segmentation Benchmarkcited by 4
  11. 2024Scaling nnU-Net for CBCT Segmentationcited by 4
  12. 2023Med-Tuning: A New Parameter-Efficient Tuning Framework for Medical Volumetric Segmentationcited by 3
  13. 2024One model to use them all: Training a segmentation model with complementary datasetscited by 3
  14. 2026The autoPET3 Challenge: Automated Lesion Segmentation in Whole-Body PET/CT unicodex2013 Multitracer Multicenter Generalizationcited by 3
  15. 2024Quality Sentinel: Estimating Label Quality and Errors in Medical Segmentation Datasetscited by 2
  16. 2025A Reality Check of Vision-Language Pre-training in Radiology: Have We Progressed Using Text?cited by 2
  17. 2025Generalist Models in Medical Image Segmentation: A Survey and Performance Comparison with Task-Specific Approachescited by 2
  18. 2025The Missing Piece: A Case for Pre-Training in 3D Medical Object Detectioncited by 2
  19. 2025BackSplit: The Importance of Sub-dividing the Background in Biomedical Lesion Segmentationcited by 2
  20. 2024Mitigating False Predictions In Unreasonable Body Regionscited by 1
  21. 2024Mask the Unknown: Assessing Different Strategies to Handle Weak Annotations in the MICCAI2023 Mediastinal Lymph Node Quantification Challengecited by 1
  22. 2025A Unified Framework for Foreground and Anonymization Area Segmentation in CT and MRI Datacited by 1
  23. 2025Towards Interactive Lesion Segmentation in Whole-Body PET/CT with Promptable Modelscited by 1
  24. 2025Progressive Growing of Patch Size: Curriculum Learning for Accelerated and Improved Medical Image Segmentationcited by 1
  25. 2024A Segmentation Foundation Model for Diverse-type Tumorscited by 0
  26. 2025Large Scale Supervised Pretraining For Traumatic Brain Injury Segmentationcited by 0
  27. 2025Exploiting Unlabeled Structures through Task Consistency Training for Versatile Medical Image Segmentationcited by 0
  28. 2025DuPLUS: Dual-Prompt Vision-Language Framework for Universal Medical Image Segmentation and Prognosiscited by 0
  29. 2025In search of truth: Evaluating concordance of AI-based anatomy segmentation modelscited by 0
  30. 2026U-Harmony: Enhancing Joint Training for Segmentation Models with Universal Harmonizationcited by 0
  31. 2026GLOW-FDG: Generalized cancer LesiOn Whole-body segmentation model for 18F-FDG-PET/CTcited by 0
  32. 2026ImputeECG: Deep Learning Reconstruction of Complete 12-Lead Electrocardiograms from Incomplete Recordings for Cardiac Assessmentcited by 0
  33. 2026What to Preserve, Where to Adapt: A Depth-Wise Analysis of Forgetting in Continual Gynecological Image Segmentationcited by 0