# SuPer Deep: A Surgical Perception Framework for Robotic Tissue Manipulation using Deep Learning for Feature Extraction

arXiv: https://arxiv.org/abs/2003.03472 · HTML: https://mlsanity.com/paper/2003.03472/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Mar 2020.

## Built on by (most cited first, 28 of 28)

- [Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery](https://mlsanity.com/paper/2206.15255/) (2022)
- [E-DSSR: Efficient Dynamic Surgical Scene Reconstruction with Transformer-based Stereoscopic Depth Perception](https://mlsanity.com/paper/2107.00229/) (2021)
- [Tracking and Mapping in Medical Computer Vision: A Review](https://mlsanity.com/paper/2310.11475/) (2023)
- [A Survey of Embodied Learning for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2408.11537/) (2024)
- [EndoGS: Deformable Endoscopic Tissues Reconstruction with Gaussian Splatting](https://mlsanity.com/paper/2401.11535/) (2024)
- [LGS: A Light-weight 4D Gaussian Splatting for Efficient Surgical Scene Reconstruction](https://mlsanity.com/paper/2406.16073/) (2024)
- [Semantic-SuPer: A Semantic-aware Surgical Perception Framework for Endoscopic Tissue Identification, Reconstruction, and Tracking](https://mlsanity.com/paper/2210.16674/) (2022)
- [SENDD: Sparse Efficient Neural Depth and Deformation for Tissue Tracking](https://mlsanity.com/paper/2305.06477/) (2023)
- [SurgicalGaussian: Deformable 3D Gaussians for High-Fidelity Surgical Scene Reconstruction](https://mlsanity.com/paper/2407.05023/) (2024)
- [Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Clouds](https://mlsanity.com/paper/2110.04685/) (2021)
- [Image-based Pose Estimation and Shape Reconstruction for Robot Manipulators and Soft, Continuum Robots via Differentiable Rendering](https://mlsanity.com/paper/2302.14039/) (2023)
- [Surgical Gym: A high-performance GPU-based platform for reinforcement learning with surgical robots](https://mlsanity.com/paper/2310.04676/) (2023)
- [Markerless Suture Needle 6D Pose Tracking with Robust Uncertainty Estimation for Autonomous Minimally Invasive Robotic Surgery](https://mlsanity.com/paper/2109.12722/) (2021)
- [DeformerNet: Learning Bimanual Manipulation of 3D Deformable Objects](https://mlsanity.com/paper/2305.04449/) (2023)
- [Robust Surgical Tool Tracking with Pixel-based Probabilities for Projected Geometric Primitives](https://mlsanity.com/paper/2403.04971/) (2024)
- [Realistic Data Generation for 6D Pose Estimation of Surgical Instruments](https://mlsanity.com/paper/2406.07328/) (2024)
- [A comprehensive survey on recent deep learning-based methods applied to surgical data](https://mlsanity.com/paper/2209.01435/) (2022)
- [DefGoalNet: Contextual Goal Learning from Demonstrations For Deformable Object Manipulation](https://mlsanity.com/paper/2309.14463/) (2023)
- [Efficient Surgical Robotic Instrument Pose Reconstruction in Real World Conditions Using Unified Feature Detection](https://mlsanity.com/paper/2510.03532/) (2025)
- [Real-Time Constrained 6D Object-Pose Tracking of An In-Hand Suture Needle for Minimally Invasive Robotic Surgery](https://mlsanity.com/paper/2210.11973/) (2022)
- [SuPerPM: A Surgical Perception Framework Based on Deep Point Matching Learned from Physical Constrained Simulation Data](https://mlsanity.com/paper/2309.13863/) (2023)
- [SurgPose: Generalisable Surgical Instrument Pose Estimation using Zero-Shot Learning and Stereo Vision](https://mlsanity.com/paper/2505.11439/) (2025)
- [Predicting the Timing of Camera Movements From the Kinematics of Instruments in Robotic-Assisted Surgery Using Artificial Neural Networks](https://mlsanity.com/paper/2109.11192/) (2021)
- [Real-time Rendering-based Surgical Instrument Tracking via Evolutionary Optimization](https://mlsanity.com/paper/2603.11404/) (2026)
- [Robustness Evaluation of Machine Learning Models for Robot Arm Action Recognition in Noisy Environments](https://mlsanity.com/paper/2401.09606/) (2024)
- [Leveraging Foundation Models To learn the shape of semi-fluid deformable objects](https://mlsanity.com/paper/2411.16802/) (2024)
- [Feature-EndoGaussian: Feature Distilled Gaussian Splatting in Surgical Deformable Scene Reconstruction](https://mlsanity.com/paper/2503.06161/) (2025)
- [The Dresden Dataset for 4D Reconstruction of Non-Rigid Abdominal Surgical Scenes](https://mlsanity.com/paper/2603.02985/) (2026)

## Abstract (arXiv)

Robotic automation in surgery requires precise tracking of surgical tools and mapping of deformable tissue. Previous works on surgical perception frameworks require significant effort in developing features for surgical tool and tissue tracking. In this work, we overcome the challenge by exploiting deep learning methods for surgical perception. We integrated deep neural networks, capable of efficient feature extraction, into the tissue reconstruction and instrument pose estimation processes. By leveraging transfer learning, the deep learning based approach requires minimal training data and reduced feature engineering efforts to fully perceive a surgical scene. The framework was tested on three publicly available datasets, which use the da Vinci Surgical System, for comprehensive analysis. Experimental results show that our framework achieves state-of-the-art tracking performance in a surgical environment by utilizing deep learning for feature extraction.