# SegPrompt: Using Segmentation Map as a Better Prompt to Finetune Deep Models for Kidney Stone Classification

arXiv: https://arxiv.org/abs/2303.08303 · HTML: https://mlsanity.com/paper/2303.08303/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 29 references that are in the map.
- First posted on arXiv 15 Mar 2023.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning](https://mlsanity.com/paper/2207.12819/) (2022)
- [Unified Vision and Language Prompt Learning](https://mlsanity.com/paper/2210.07225/) (2022)
- [Neural Prompt Search](https://mlsanity.com/paper/2206.04673/) (2022)
- [Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study](https://mlsanity.com/paper/2209.15517/) (2022)
- [SkinCon: A skin disease dataset densely annotated by domain experts for fine-grained model debugging and analysis](https://mlsanity.com/paper/2302.00785/) (2023)
- [Towards Automatic Recognition of Pure & Mixed Stones using Intraoperative Endoscopic Digital Images](https://mlsanity.com/paper/2105.10686/) (2021)
- [On the in vivo recognition of kidney stones using machine learning](https://mlsanity.com/paper/2201.08865/) (2022)
- [Alleviating the Incompatibility between Cross Entropy Loss and Episode Training for Few-shot Skin Disease Classification](https://mlsanity.com/paper/2004.09694/) (2020)

## Built on by (most cited first, 4 of 4)

- [Prompt Engineering for Healthcare: Methodologies and Applications](https://mlsanity.com/paper/2304.14670/) (2023)
- [Causal Scoring Medical Image Explanations: A Case Study On Ex-vivo Kidney Stone Images](https://mlsanity.com/paper/2309.01921/) (2023)
- [Prompt Mechanisms in Medical Imaging: A Comprehensive Survey](https://mlsanity.com/paper/2507.01055/) (2025)
- [USCNet: Transformer-Based Multimodal Fusion with Segmentation Guidance for Urolithiasis Classification](https://mlsanity.com/paper/2604.07141/) (2026)

## Abstract (arXiv)

Recently, deep learning has produced encouraging results for kidney stone classification using endoscope images. However, the shortage of annotated training data poses a severe problem in improving the performance and generalization ability of the trained model. It is thus crucial to fully exploit the limited data at hand. In this paper, we propose SegPrompt to alleviate the data shortage problems by exploiting segmentation maps from two aspects. First, SegPrompt integrates segmentation maps to facilitate classification training so that the classification model is aware of the regions of interest. The proposed method allows the image and segmentation tokens to interact with each other to fully utilize the segmentation map information. Second, we use the segmentation maps as prompts to tune the pretrained deep model, resulting in much fewer trainable parameters than vanilla finetuning. We perform extensive experiments on the collected kidney stone dataset. The results show that SegPrompt can achieve an advantageous balance between the model fitting ability and the generalization ability, eventually leading to an effective model with limited training data.