# Boosting Medical Image Classification with Segmentation Foundation Model

arXiv: https://arxiv.org/abs/2406.11026 · HTML: https://mlsanity.com/paper/2406.11026/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 16 Jun 2024.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [Customized Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.13785/) (2023)
- [Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis](https://mlsanity.com/paper/1912.08937/) (2019)
- [Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imaging](https://mlsanity.com/paper/2304.04155/) (2023)
- [Polyp-SAM: Transfer SAM for Polyp Segmentation](https://mlsanity.com/paper/2305.00293/) (2023)
- [Input Augmentation with SAM: Boosting Medical Image Segmentation with Segmentation Foundation Model](https://mlsanity.com/paper/2304.11332/) (2023)
- [Context-Aware Convolutional Neural Network for Grading of Colorectal Cancer Histology Images](https://mlsanity.com/paper/1907.09478/) (2019)
- [Data-Driven Color Augmentation Techniques for Deep Skin Image Analysis](https://mlsanity.com/paper/1703.03702/) (2017)
- [Incorporating the Knowledge of Dermatologists to Convolutional Neural Networks for the Diagnosis of Skin Lesions](https://mlsanity.com/paper/1703.01976/) (2017)
- [Data-Driven Deep Supervision for Skin Lesion Classification](https://mlsanity.com/paper/2209.01527/) (2022)

## Built on by (most cited first, 8 of 8)

- [SAUGE: Taming SAM for Uncertainty-Aligned Multi-Granularity Edge Detection](https://mlsanity.com/paper/2412.12892/) (2024)
- [Topo-VM-UNetV2: Encoding Topology into Vision Mamba UNet for Polyp Segmentation](https://mlsanity.com/paper/2505.06210/) (2025)
- [TopoImages: Incorporating Local Topology Encoding into Deep Learning Models for Medical Image Classification](https://mlsanity.com/paper/2508.01574/) (2025)
- [Integrating Multi-scale and Multi-filtration Topological Features for Medical Image Classification](https://mlsanity.com/paper/2512.07190/) (2025)
- [Semi-Supervised Medical Image Segmentation via Knowledge Mining from Large Models](https://mlsanity.com/paper/2503.06816/) (2025)
- [White Light Specular Reflection Data Augmentation for Deep Learning Polyp Detection](https://mlsanity.com/paper/2505.05248/) (2025)
- [Adapting a Segmentation Foundation Model for Medical Image Classification](https://mlsanity.com/paper/2505.06217/) (2025)
- [Prompt-Free SAM-Based Multi-Task Framework for Breast Ultrasound Lesion Segmentation and Classification](https://mlsanity.com/paper/2601.05498/) (2026)

## Abstract (arXiv)

The Segment Anything Model (SAM) exhibits impressive capabilities in zero-shot segmentation for natural images. Recently, SAM has gained a great deal of attention for its applications in medical image segmentation. However, to our best knowledge, no studies have shown how to harness the power of SAM for medical image classification. To fill this gap and make SAM a true ``foundation model'' for medical image analysis, it is highly desirable to customize SAM specifically for medical image classification. In this paper, we introduce SAMAug-C, an innovative augmentation method based on SAM for augmenting classification datasets by generating variants of the original images. The augmented datasets can be used to train a deep learning classification model, thereby boosting the classification performance. Furthermore, we propose a novel framework that simultaneously processes raw and SAMAug-C augmented image input, capitalizing on the complementary information that is offered by both. Experiments on three public datasets validate the effectiveness of our new approach.