# Rethinking model prototyping through the MedMNIST+ dataset collection

arXiv: https://arxiv.org/abs/2404.15786 · HTML: https://mlsanity.com/paper/2404.15786/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 67 references that are in the map.
- First posted on arXiv 24 Apr 2024.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [Learning to Generalize: Meta-Learning for Domain Generalization](https://mlsanity.com/paper/1710.03463/) (2017)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [The Liver Tumor Segmentation Benchmark (LiTS)](https://mlsanity.com/paper/1901.04056/) (2019)
- [EVA-02: A Visual Representation for Neon Genesis](https://mlsanity.com/paper/2303.11331/) (2023)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Inductive Biases for Deep Learning of Higher-Level Cognition](https://mlsanity.com/paper/2011.15091/) (2020)
- [Large image datasets: A pyrrhic win for computer vision?](https://mlsanity.com/paper/2006.16923/) (2020)
- [Compute Trends Across Three Eras of Machine Learning](https://mlsanity.com/paper/2202.05924/) (2022)
- [Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research](https://mlsanity.com/paper/2112.01716/) (2021)
- [Delving Deep into the Generalization of Vision Transformers under Distribution Shifts](https://mlsanity.com/paper/2106.07617/) (2021)
- [Learning Invariant Representations and Risks for Semi-supervised Domain Adaptation](https://mlsanity.com/paper/2010.04647/) (2020)
- [Detecting Corrupted Labels Without Training a Model to Predict](https://mlsanity.com/paper/2110.06283/) (2021)
- [Machine learning and deep learning](https://mlsanity.com/paper/2104.05314/) (2021)
- [Revisiting a kNN-based Image Classification System with High-capacity Storage](https://mlsanity.com/paper/2204.01186/) (2022)
- [Deep Learning Based Detection and Correction of Cardiac MR Motion Artefacts During Reconstruction for High-Quality Segmentation](https://mlsanity.com/paper/1910.05370/) (2019)
- [Distilling Out-of-Distribution Robustness from Vision-Language Foundation Models](https://mlsanity.com/paper/2311.01441/) (2023)
- [Integrating kNN with Foundation Models for Adaptable and Privacy-Aware Image Classification](https://mlsanity.com/paper/2402.12500/) (2024)

## Built on by (most cited first, 9 of 9)

- [Privacy-preserving datasets by capturing feature distributions with Conditional VAEs](https://mlsanity.com/paper/2408.00639/) (2024)
- [Toward Aristotelian Medical Representations: Backpropagation-Free Layer-wise Analysis for Interpretable Generalized Metric Learning on MedMNIST](https://mlsanity.com/paper/2604.06017/) (2026)
- [HypCBC: Domain-Invariant Hyperbolic Cross-Branch Consistency for Generalizable Medical Image Analysis](https://mlsanity.com/paper/2602.03264/) (2026)
- [Unmasking Biases and Reliability Concerns in Convolutional Neural Networks Analysis of Cancer Pathology Images](https://mlsanity.com/paper/2603.12445/) (2026)
- [A Self supervised learning framework for imbalanced medical imaging datasets](https://mlsanity.com/paper/2604.01947/) (2026)
- [LiLAW: Lightweight Learnable Adaptive Weighting to Learn Sample Difficulty & Improve Noisy Training](https://mlsanity.com/paper/2509.20786/) (2025)
- [Benchmarking PNW Model for MedMNIST to 100% Accuracy](https://mlsanity.com/paper/2604.18916/) (2026)
- [MoPET: Parameter-Efficient Mixture-of-Experts for Unified Medical Image Classification](https://mlsanity.com/paper/2607.29462/) (2026)
- [A Shared-Backbone Approach for Multi-Task MedMNIST Classification](https://mlsanity.com/paper/2609.06838/) (2026)

## Abstract (arXiv)

The integration of deep learning based systems in clinical practice is often impeded by challenges rooted in limited and heterogeneous medical datasets. In addition, the field has increasingly prioritized marginal performance gains on a few, narrowly scoped benchmarks over clinical applicability, slowing down meaningful algorithmic progress. This trend often results in excessive fine-tuning of existing methods on selected datasets rather than fostering clinically relevant innovations. In response, this work introduces a comprehensive benchmark for the MedMNIST+ dataset collection, designed to diversify the evaluation landscape across several imaging modalities, anatomical regions, classification tasks and sample sizes. We systematically reassess commonly used Convolutional Neural Networks (CNNs) and Vision Transformer (ViT) architectures across distinct medical datasets, training methodologies, and input resolutions to validate and refine existing assumptions about model effectiveness and development. Our findings suggest that computationally efficient training schemes and modern foundation models offer viable alternatives to costly end-to-end training. Additionally, we observe that higher image resolutions do not consistently improve performance beyond a certain threshold. This highlights the potential benefits of using lower resolutions, particularly in prototyping stages, to reduce computational demands without sacrificing accuracy. Notably, our analysis reaffirms the competitiveness of CNNs compared to ViTs, emphasizing the importance of comprehending the intrinsic capabilities of different architectures. Finally, by establishing a standardized evaluation framework, we aim to enhance transparency, reproducibility, and comparability within the MedMNIST+ dataset collection. Code is available at https://github.com/sdoerrich97/rethinking-model-prototyping-MedMNISTPlus .