# Multi-class motion-based semantic segmentation for ureteroscopy and laser lithotripsy

arXiv: https://arxiv.org/abs/2104.01268 · HTML: https://mlsanity.com/paper/2104.01268/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 56 references that are in the map.
- First posted on arXiv 2 Apr 2021.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Road Extraction by Deep Residual U-Net](https://mlsanity.com/paper/1711.10684/) (2017)
- [ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data](https://mlsanity.com/paper/1904.00592/) (2019)
- [ResUNet++: An Advanced Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1911.07067/) (2019)
- [Recurrent Residual Convolutional Neural Network based on U-Net (R2U-Net) for Medical Image Segmentation](https://mlsanity.com/paper/1802.06955/) (2018)
- [The VIA Annotation Software for Images, Audio and Video](https://mlsanity.com/paper/1904.10699/) (2019)
- [A Deep Learning Framework for Unsupervised Affine and Deformable Image Registration](https://mlsanity.com/paper/1809.06130/) (2018)
- [Quicksilver: Fast Predictive Image Registration - a Deep Learning Approach](https://mlsanity.com/paper/1703.10908/) (2017)
- [Joint Learning of Motion Estimation and Segmentation for Cardiac MR Image Sequences](https://mlsanity.com/paper/1806.04066/) (2018)
- [Inverse-Consistent Deep Networks for Unsupervised Deformable Image Registration](https://mlsanity.com/paper/1809.03443/) (2018)
- [Boundary Loss for Remote Sensing Imagery Semantic Segmentation](https://mlsanity.com/paper/1905.07852/) (2019)
- [Effective Use of Dilated Convolutions for Segmenting Small Object Instances in Remote Sensing Imagery](https://mlsanity.com/paper/1709.00179/) (2017)
- [Real-Time Polyp Detection, Localization and Segmentation in Colonoscopy Using Deep Learning](https://mlsanity.com/paper/2011.07631/) (2020)
- [OR-UNet: an Optimized Robust Residual U-Net for Instrument Segmentation in Endoscopic Images](https://mlsanity.com/paper/2004.12668/) (2020)
- [Polyp Detection and Segmentation from Video Capsule Endoscopy: A Review](https://mlsanity.com/paper/1609.01915/) (2016)
- [Conv2Warp: An unsupervised deformable image registration with continuous convolution and warping](https://mlsanity.com/paper/1908.06194/) (2019)
- [Deep learning for detection and segmentation of artefact and disease instances in gastrointestinal endoscopy](https://mlsanity.com/paper/2010.06034/) (2020)

## Built on by (most cited first, 1 of 1)

- [Assessing the generalization performance of SAM for ureteroscopy scene understanding](https://mlsanity.com/paper/2505.17210/) (2025)

## Abstract (arXiv)

Kidney stones represent a considerable burden for public health-care systems. Ureteroscopy with laser lithotripsy has evolved as the most commonly used technique for the treatment of kidney stones. Automated segmentation of kidney stones and laser fiber is an important initial step to performing any automated quantitative analysis of the stones, particularly stone-size estimation, that helps the surgeon decide if the stone requires more fragmentation. Factors such as turbid fluid inside the cavity, specularities, motion blur due to kidney movements and camera motion, bleeding, and stone debris impact the quality of vision within the kidney and lead to extended operative times. To the best of our knowledge, this is the first attempt made towards multi-class segmentation in ureteroscopy and laser lithotripsy data. We propose an end-to-end CNN-based framework for the segmentation of stones and laser fiber. The proposed approach utilizes two sub-networks: HybResUNet, a version of residual U-Net, that uses residual connections in the encoder path of U-Net and a DVFNet that generates DVF predictions which are then used to prune the prediction maps. We also present ablation studies that combine dilated convolutions, recurrent and residual connections, ASPP and attention gate. We propose a compound loss function that improves our segmentation performance. We have also provided an ablation study to determine the optimal data augmentation strategy. Our qualitative and quantitative results illustrate that our proposed method outperforms SOTA methods such as UNet and DeepLabv3+ showing an improvement of 5.2% and 15.93%, respectively, for the combined mean of DSC and JI in our invivo test dataset. We also show that our proposed model generalizes better on a new clinical dataset showing a mean improvement of 25.4%, 20%, and 11% over UNet, HybResUNet, and DeepLabv3+, respectively, for the same metric.