# Using spatial-temporal ensembles of convolutional neural networks for lumen segmentation in ureteroscopy

arXiv: https://arxiv.org/abs/2104.01985 · HTML: https://mlsanity.com/paper/2104.01985/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 23 references that are in the map.
- First posted on arXiv 5 Apr 2021.

## Builds on (most cited first, 7 of 7)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [A Benchmark for Endoluminal Scene Segmentation of Colonoscopy Images](https://mlsanity.com/paper/1612.00799/) (2016)
- [Mask-RCNN and U-net Ensembled for Nuclei Segmentation](https://mlsanity.com/paper/1901.10170/) (2019)
- [A New Ensemble Learning Framework for 3D Biomedical Image Segmentation](https://mlsanity.com/paper/1812.03945/) (2018)
- [Preterm infants' pose estimation with spatio-temporal features](https://mlsanity.com/paper/2005.08648/) (2020)
- [A Lumen Segmentation Method in Ureteroscopy Images based on a Deep Residual U-Net architecture](https://mlsanity.com/paper/2101.05021/) (2021)

## Built on by (most cited first, 1 of 1)

- [Autonomous Intraluminal Navigation of a Soft Robot using Deep-Learning-based Visual Servoing](https://mlsanity.com/paper/2207.00401/) (2022)

## Abstract (arXiv)

Purpose: Ureteroscopy is an efficient endoscopic minimally invasive technique for the diagnosis and treatment of upper tract urothelial carcinoma (UTUC). During ureteroscopy, the automatic segmentation of the hollow lumen is of primary importance, since it indicates the path that the endoscope should follow. In order to obtain an accurate segmentation of the hollow lumen, this paper presents an automatic method based on Convolutional Neural Networks (CNNs). Methods: The proposed method is based on an ensemble of 4 parallel CNNs to simultaneously process single and multi-frame information. Of these, two architectures are taken as core-models, namely U-Net based in residual blocks(m1) and Mask-RCNN(m2), which are fed with single still-frames I(t). The other two models (M1, M2) are modifications of the former ones consisting on the addition of a stage which makes use of 3D Convolutions to process temporal information. M1, M2 are fed with triplets of frames (I(t-1), I(t), I(t+1)) to produce the segmentation for I(t). Results: The proposed method was evaluated using a custom dataset of 11 videos (2,673 frames) which were collected and manually annotated from 6 patients. We obtain a Dice similarity coefficient of 0.80, outperforming previous state-of-the-art methods. Conclusion: The obtained results show that spatial-temporal information can be effectively exploited by the ensemble model to improve hollow lumen segmentation in ureteroscopic images. The method is effective also in presence of poor visibility, occasional bleeding, or specular reflections.