# Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?

arXiv: https://arxiv.org/abs/2411.03670 · HTML: https://mlsanity.com/paper/2411.03670/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 92 references that are in the map.
- First posted on arXiv 6 Nov 2024.

## Builds on (most cited first, 30 of 63)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [A Reductions Approach to Fair Classification](https://mlsanity.com/paper/1803.02453/) (2018)
- [UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation](https://mlsanity.com/paper/1912.05074/) (2019)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [The Liver Tumor Segmentation Benchmark (LiTS)](https://mlsanity.com/paper/1901.04056/) (2019)
- [MONAI: An open-source framework for deep learning in healthcare](https://mlsanity.com/paper/2211.02701/) (2022)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation](https://mlsanity.com/paper/2404.09556/) (2024)
- [AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?](https://mlsanity.com/paper/2010.14808/) (2020)
- [CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection](https://mlsanity.com/paper/2301.00785/) (2023)
- [UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer](https://mlsanity.com/paper/2109.04335/) (2021)
- [The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 Challenge](https://mlsanity.com/paper/1912.01054/) (2019)
- [Are Transformers More Robust Than CNNs?](https://mlsanity.com/paper/2111.05464/) (2021)
- [Vision GNN: An Image is Worth Graph of Nodes](https://mlsanity.com/paper/2206.00272/) (2022)
- [The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomes](https://mlsanity.com/paper/1904.00445/) (2019)
- [MedNeXt: Transformer-driven Scaling of ConvNets for Medical Image Segmentation](https://mlsanity.com/paper/2303.09975/) (2023)
- [M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models](https://mlsanity.com/paper/2404.00578/) (2024)
- [DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentation](https://mlsanity.com/paper/1506.06448/) (2015)

## Built on by (most cited first, 30 of 31)

- [nnInteractive: Redefining 3D Promptable Segmentation](https://mlsanity.com/paper/2503.08373/) (2025)
- [RadGPT: Constructing 3D Image-Text Tumor Datasets](https://mlsanity.com/paper/2501.04678/) (2025)
- [An OpenMind for 3D medical vision self-supervised learning](https://mlsanity.com/paper/2412.17041/) (2024)
- [Primus: Enforcing Attention Usage for 3D Medical Image Segmentation](https://mlsanity.com/paper/2503.01835/) (2025)
- [PanTS: The Pancreatic Tumor Segmentation Dataset](https://mlsanity.com/paper/2507.01291/) (2025)
- [LesionLocator: Zero-Shot Universal Tumor Segmentation and Tracking in 3D Whole-Body Imaging](https://mlsanity.com/paper/2502.20985/) (2025)
- [Learning Segmentation from Radiology Reports](https://mlsanity.com/paper/2507.05582/) (2025)
- [UniBiomed: A Universal Foundation Model for Grounded Biomedical Image Interpretation](https://mlsanity.com/paper/2504.21336/) (2025)
- [Scaling Artificial Intelligence for Multi-Tumor Early Detection with More Reports, Fewer Masks](https://mlsanity.com/paper/2510.14803/) (2025)
- [Revisiting 2D Foundation Models for Scalable 3D Medical Image Classification](https://mlsanity.com/paper/2512.12887/) (2025)
- [Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS) challenge results](https://mlsanity.com/paper/2505.08685/) (2025)
- [Are Pixel-Wise Metrics Reliable for Sparse-View Computed Tomography Reconstruction?](https://mlsanity.com/paper/2506.02093/) (2025)
- [Expectation-Maximization as the Engine of Scalable Medical Intelligence](https://mlsanity.com/paper/2501.03410/) (2025)
- [MedNeXt-v2: Scaling 3D ConvNeXts for Large-Scale Supervised Representation Learning in Medical Image Segmentation](https://mlsanity.com/paper/2512.17774/) (2025)
- [Efficient MedSAMs: Segment Anything in Medical Images on Laptop](https://mlsanity.com/paper/2412.16085/) (2024)
- [RadThinking: A Dataset for Longitudinal Clinical Reasoning in Radiology](https://mlsanity.com/paper/2605.10761/) (2026)
- [Automating RT Planning at Scale: High Quality Data For AI Training](https://mlsanity.com/paper/2501.11803/) (2025)
- [See More, Change Less: Anatomy-Aware Diffusion for Contrast Enhancement](https://mlsanity.com/paper/2512.07251/) (2025)
- [ENSAM: an efficient foundation model for interactive segmentation of 3D medical images](https://mlsanity.com/paper/2509.15874/) (2025)
- [Auditing Significance, Metric Choice, and Demographic Fairness in Medical AI Challenges](https://mlsanity.com/paper/2512.19091/) (2025)
- [BiPVL-Seg: Bidirectional Progressive Vision-Language Fusion with Global-Local Alignment for Medical Image Segmentation](https://mlsanity.com/paper/2503.23534/) (2025)
- [Rethink Domain Generalization in Heterogeneous Sequence MRI Segmentation](https://mlsanity.com/paper/2507.23110/) (2025)
- [CardioBench: Do Echocardiography Foundation Models Generalize Beyond the Lab?](https://mlsanity.com/paper/2510.00520/) (2025)
- [Universality Reconsidered: Rethinking the Validation of Foundation Models for General-Purpose 3D Medical Segmentation](https://mlsanity.com/paper/2602.07643/) (2026)
- [Random Window Augmentations for Deep Learning Robustness in CT and Liver Tumor Segmentation](https://mlsanity.com/paper/2510.08116/) (2025)
- [Knowledge Transfer Scaling Laws for 3D Medical Imaging](https://mlsanity.com/paper/2605.06859/) (2026)
- [Beyond Masks: The Case for Medical Image Parsing](https://mlsanity.com/paper/2605.11438/) (2026)
- [POPSICLE: Benchmark Datasets for Segmentation and Localization in CryoET](https://mlsanity.com/paper/2606.10255/) (2026)
- [Report Supervision](https://mlsanity.com/paper/2608.27668/) (2026)
- [RT-Super: Learning Tumor Segmentation from Longitudinal Images and Reports](https://mlsanity.com/paper/2609.35637/) (2026)

## Abstract (arXiv)

How can we test AI performance? This question seems trivial, but it isn't. Standard benchmarks often have problems such as in-distribution and small-size test sets, oversimplified metrics, unfair comparisons, and short-term outcome pressure. As a consequence, good performance on standard benchmarks does not guarantee success in real-world scenarios. To address these problems, we present Touchstone, a large-scale collaborative segmentation benchmark of 9 types of abdominal organs. This benchmark is based on 5,195 training CT scans from 76 hospitals around the world and 5,903 testing CT scans from 11 additional hospitals. This diverse test set enhances the statistical significance of benchmark results and rigorously evaluates AI algorithms across various out-of-distribution scenarios. We invited 14 inventors of 19 AI algorithms to train their algorithms, while our team, as a third party, independently evaluated these algorithms on three test sets. In addition, we also evaluated pre-existing AI frameworks--which, differing from algorithms, are more flexible and can support different algorithms--including MONAI from NVIDIA, nnU-Net from DKFZ, and numerous other open-source frameworks. We are committed to expanding this benchmark to encourage more innovation of AI algorithms for the medical domain.