# Overfitting in Histopathology Model Training: The Need for Customized Architectures

arXiv: https://arxiv.org/abs/2506.16631 · HTML: https://mlsanity.com/paper/2506.16631/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 15 references that are in the map.
- First posted on arXiv 19 Jun 2025.

## Builds on (most cited first, 7 of 7)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A scoping review of transfer learning research on medical image analysis using ImageNet](https://mlsanity.com/paper/2004.13175/) (2020)
- [Fine-Tuning and Training of DenseNet for Histopathology Image Representation Using TCGA Diagnostic Slides](https://mlsanity.com/paper/2101.07903/) (2021)
- [Rotation-Agnostic Image Representation Learning for Digital Pathology](https://mlsanity.com/paper/2311.08359/) (2023)
- [Improving Feature Extraction from Histopathological Images Through A Fine-tuning ImageNet Model](https://mlsanity.com/paper/2201.00636/) (2022)
- [HistoTransfer: Understanding Transfer Learning for Histopathology](https://mlsanity.com/paper/2106.07068/) (2021)

## Abstract (arXiv)

This study investigates the critical problem of overfitting in deep learning models applied to histopathology image analysis. We show that simply adopting and fine-tuning large-scale models designed for natural image analysis often leads to suboptimal performance and significant overfitting when applied to histopathology tasks. Through extensive experiments with various model architectures, including ResNet variants and Vision Transformers (ViT), we show that increasing model capacity does not necessarily improve performance on histopathology datasets. Our findings emphasize the need for customized architectures specifically designed for histopathology image analysis, particularly when working with limited datasets. Using Oesophageal Adenocarcinomas public dataset, we demonstrate that simpler, domain-specific architectures can achieve comparable or better performance while minimizing overfitting.