# A Kinematic Bottleneck Approach For Pose Regression of Flexible Surgical Instruments directly from Images

arXiv: https://arxiv.org/abs/2103.00586 · HTML: https://mlsanity.com/paper/2103.00586/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 44 references that are in the map.
- First posted on arXiv 28 Feb 2021.

## Builds on (most cited first, 19 of 19)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Generating Images with Perceptual Similarity Metrics based on Deep Networks](https://mlsanity.com/paper/1602.02644/) (2016)
- [Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild](https://mlsanity.com/paper/1911.11130/) (2019)
- [Surgical Data Science: Enabling Next-Generation Surgery](https://mlsanity.com/paper/1701.06482/) (2017)
- [Image inpainting: A review](https://mlsanity.com/paper/1909.06399/) (2019)
- [Learning from Synthetic Animals](https://mlsanity.com/paper/1912.08265/) (2019)
- [Super-Resolution with Deep Convolutional Sufficient Statistics](https://mlsanity.com/paper/1511.05666/) (2015)
- [Weakly Supervised Convolutional LSTM Approach for Tool Tracking in Laparoscopic Videos](https://mlsanity.com/paper/1812.01366/) (2018)
- [Exploiting the potential of unlabeled endoscopic video data with self-supervised learning](https://mlsanity.com/paper/1711.09726/) (2017)
- [Concurrent Segmentation and Localization for Tracking of Surgical Instruments](https://mlsanity.com/paper/1703.10701/) (2017)
- [Learning from a tiny dataset of manual annotations: a teacher/student approach for surgical phase recognition](https://mlsanity.com/paper/1812.00033/) (2018)
- [Self-Supervised Surgical Tool Segmentation using Kinematic Information](https://mlsanity.com/paper/1902.04810/) (2019)
- [Simultaneous Recognition and Pose Estimation of Instruments in Minimally Invasive Surgery](https://mlsanity.com/paper/1710.06668/) (2017)
- [Motion2Vec: Semi-Supervised Representation Learning from Surgical Videos](https://mlsanity.com/paper/2006.00545/) (2020)
- [Towards Unsupervised Learning for Instrument Segmentation in Robotic Surgery with Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/2007.04505/) (2020)

## Built on by (most cited first, 11 of 11)

- [Dissecting Self-Supervised Learning Methods for Surgical Computer Vision](https://mlsanity.com/paper/2207.00449/) (2022)
- [TraSeTR: Track-to-Segment Transformer with Contrastive Query for Instance-level Instrument Segmentation in Robotic Surgery](https://mlsanity.com/paper/2202.08453/) (2022)
- [Self-Supervised Learning for Endoscopic Video Analysis](https://mlsanity.com/paper/2308.12394/) (2023)
- [FUN-SIS: a Fully UNsupervised approach for Surgical Instrument Segmentation](https://mlsanity.com/paper/2202.08141/) (2022)
- [Jumpstarting Surgical Computer Vision](https://mlsanity.com/paper/2312.05968/) (2023)
- [Reducing Annotating Load: Active Learning with Synthetic Images in Surgical Instrument Segmentation](https://mlsanity.com/paper/2108.03534/) (2021)
- [Personalizing Federated Instrument Segmentation with Visual Trait Priors in Robotic Surgery](https://mlsanity.com/paper/2408.03208/) (2024)
- [SAF-IS: a Spatial Annotation Free Framework for Instance Segmentation of Surgical Tools](https://mlsanity.com/paper/2309.01723/) (2023)
- [Visual-Kinematics Graph Learning for Procedure-agnostic Instrument Tip Segmentation in Robotic Surgeries](https://mlsanity.com/paper/2309.00957/) (2023)
- [SurgPIS: Surgical-instrument-level Instances and Part-level Semantics for Weakly-supervised Part-aware Instance Segmentation](https://mlsanity.com/paper/2507.19592/) (2025)
- [TEDi: Temporal Memory-Enhanced and Denoising Transformer for Surgical Instrument Segmentation](https://mlsanity.com/paper/2609.16797/) (2026)

## Abstract (arXiv)

3-D pose estimation of instruments is a crucial step towards automatic scene understanding in robotic minimally invasive surgery. Although robotic systems can potentially directly provide joint values, this information is not commonly exploited inside the operating room, due to its possible unreliability, limited access and the time-consuming calibration required, especially for continuum robots. For this reason, standard approaches for 3-D pose estimation involve the use of external tracking systems. Recently, image-based methods have emerged as promising, non-invasive alternatives. While many image-based approaches in the literature have shown accurate results, they generally require either a complex iterative optimization for each processed image, making them unsuitable for real-time applications, or a large number of manually-annotated images for efficient learning. In this paper we propose a self-supervised image-based method, exploiting, at training time only, the imprecise kinematic information provided by the robot. In order to avoid introducing time-consuming manual annotations, the problem is formulated as an auto-encoder, smartly bottlenecked by the presence of a physical model of the robotic instruments and surgical camera, forcing a separation between image background and kinematic content. Validation of the method was performed on semi-synthetic, phantom and in-vivo datasets, obtained using a flexible robotized endoscope, showing promising results for real-time image-based 3-D pose estimation of surgical instruments.