# A-Eval: A Benchmark for Cross-Dataset Evaluation of Abdominal Multi-Organ Segmentation

arXiv: https://arxiv.org/abs/2309.03906 · HTML: https://mlsanity.com/paper/2309.03906/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 55 references that are in the map.
- First posted on arXiv 7 Sep 2023.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [The Liver Tumor Segmentation Benchmark (LiTS)](https://mlsanity.com/paper/1901.04056/) (2019)
- [Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1708.08987/) (2017)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [CHAOS Challenge -- Combined (CT-MR) Healthy Abdominal Organ Segmentation](https://mlsanity.com/paper/2001.06535/) (2020)
- [Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation](https://mlsanity.com/paper/1908.10454/) (2019)
- [AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?](https://mlsanity.com/paper/2010.14808/) (2020)
- [Data-centric Artificial Intelligence: A Survey](https://mlsanity.com/paper/2303.10158/) (2023)
- [The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 Challenge](https://mlsanity.com/paper/1912.01054/) (2019)
- [The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomes](https://mlsanity.com/paper/1904.00445/) (2019)
- [DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentation](https://mlsanity.com/paper/1506.06448/) (2015)
- [MS-Net: Multi-Site Network for Improving Prostate Segmentation with Heterogeneous MRI Data](https://mlsanity.com/paper/2002.03366/) (2020)
- [The KiTS21 Challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase CT](https://mlsanity.com/paper/2307.01984/) (2023)
- [STU-Net: Scalable and Transferable Medical Image Segmentation Models Empowered by Large-Scale Supervised Pre-training](https://mlsanity.com/paper/2304.06716/) (2023)
- [Unsupervised Cross-Modality Domain Adaptation of ConvNets for Biomedical Image Segmentations with Adversarial Loss](https://mlsanity.com/paper/1804.10916/) (2018)
- [Unleashing the Strengths of Unlabeled Data in Pan-cancer Abdominal Organ Quantification: the FLARE22 Challenge](https://mlsanity.com/paper/2308.05862/) (2023)
- [Abdominal multi-organ segmentation with organ-attention networks and statistical fusion](https://mlsanity.com/paper/1804.08414/) (2018)
- [Rethinking Data Augmentation for Single-source Domain Generalization in Medical Image Segmentation](https://mlsanity.com/paper/2211.14805/) (2022)
- [The Domain Shift Problem of Medical Image Segmentation and Vendor-Adaptation by Unet-GAN](https://mlsanity.com/paper/1910.13681/) (2019)
- [The Principles of Data-Centric AI (DCAI)](https://mlsanity.com/paper/2211.14611/) (2022)
- [Organ at Risk Segmentation for Head and Neck Cancer using Stratified Learning and Neural Architecture Search](https://mlsanity.com/paper/2004.08426/) (2020)
- [Boundary-weighted Domain Adaptive Neural Network for Prostate MR Image Segmentation](https://mlsanity.com/paper/1902.08128/) (2019)
- [Embracing the Disharmony in Medical Imaging: A Simple and Effective Framework for Domain Adaptation](https://mlsanity.com/paper/2103.12857/) (2021)
- [A multi-scale pyramid of 3D fully convolutional networks for abdominal multi-organ segmentation](https://mlsanity.com/paper/1806.02237/) (2018)
- [Deep Q Learning Driven CT Pancreas Segmentation with Geometry-Aware U-Net](https://mlsanity.com/paper/1904.09120/) (2019)
- [Crossbar-Net: A Novel Convolutional Network for Kidney Tumor Segmentation in CT Images](https://mlsanity.com/paper/1804.10484/) (2018)
- [Abdominal multi-organ segmentation with cascaded convolutional and adversarial deep networks](https://mlsanity.com/paper/2001.09521/) (2020)
- [Generalizability of Machine Learning Models: Quantitative Evaluation of Three Methodological Pitfalls](https://mlsanity.com/paper/2202.01337/) (2022)

## Built on by (most cited first, 4 of 4)

- [AbdomenAtlas: A Large-Scale, Detailed-Annotated, & Multi-Center Dataset for Efficient Transfer Learning and Open Algorithmic Benchmarking](https://mlsanity.com/paper/2407.16697/) (2024)
- [Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?](https://mlsanity.com/paper/2411.03670/) (2024)
- [Data-Centric Foundation Models in Computational Healthcare: A Survey](https://mlsanity.com/paper/2401.02458/) (2024)
- [MLLA-UNet: Mamba-like Linear Attention in an Efficient U-Shape Model for Medical Image Segmentation](https://mlsanity.com/paper/2410.23738/) (2024)

## Abstract (arXiv)

Although deep learning have revolutionized abdominal multi-organ segmentation, models often struggle with generalization due to training on small, specific datasets. With the recent emergence of large-scale datasets, some important questions arise: Can models trained on these datasets generalize well on different ones? If yes/no, how to further improve their generalizability? To address these questions, we introduce A-Eval, a benchmark for the cross-dataset Evaluation ('Eval') of Abdominal ('A') multi-organ segmentation. We employ training sets from four large-scale public datasets: FLARE22, AMOS, WORD, and TotalSegmentator, each providing extensive labels for abdominal multi-organ segmentation. For evaluation, we incorporate the validation sets from these datasets along with the training set from the BTCV dataset, forming a robust benchmark comprising five distinct datasets. We evaluate the generalizability of various models using the A-Eval benchmark, with a focus on diverse data usage scenarios: training on individual datasets independently, utilizing unlabeled data via pseudo-labeling, mixing different modalities, and joint training across all available datasets. Additionally, we explore the impact of model sizes on cross-dataset generalizability. Through these analyses, we underline the importance of effective data usage in enhancing models' generalization capabilities, offering valuable insights for assembling large-scale datasets and improving training strategies. The code and pre-trained models are available at hrefhttps://github.com/uni-medical/A-Evalhttps://github.com/uni-medical/A-Eval.