# SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation with Surgical Robotics

arXiv: https://arxiv.org/abs/1909.05405 · HTML: https://mlsanity.com/paper/1909.05405/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Sep 2019.

## Built on by (most cited first, 30 of 47)

- [Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery](https://mlsanity.com/paper/2206.15255/) (2022)
- [EndoSurf: Neural Surface Reconstruction of Deformable Tissues with Stereo Endoscope Videos](https://mlsanity.com/paper/2307.11307/) (2023)
- [E-DSSR: Efficient Dynamic Surgical Scene Reconstruction with Transformer-based Stereoscopic Depth Perception](https://mlsanity.com/paper/2107.00229/) (2021)
- [Tracking and Mapping in Medical Computer Vision: A Review](https://mlsanity.com/paper/2310.11475/) (2023)
- [EndoGS: Deformable Endoscopic Tissues Reconstruction with Gaussian Splatting](https://mlsanity.com/paper/2401.11535/) (2024)
- [SurRoL: An Open-source Reinforcement Learning Centered and dVRK Compatible Platform for Surgical Robot Learning](https://mlsanity.com/paper/2108.13035/) (2021)
- [Markerless Camera-to-Robot Pose Estimation via Self-supervised Sim-to-Real Transfer](https://mlsanity.com/paper/2302.14332/) (2023)
- [EndoGaussian: Real-time Gaussian Splatting for Dynamic Endoscopic Scene Reconstruction](https://mlsanity.com/paper/2401.12561/) (2024)
- [LGS: A Light-weight 4D Gaussian Splatting for Efficient Surgical Scene Reconstruction](https://mlsanity.com/paper/2406.16073/) (2024)
- [Efficient Deformable Tissue Reconstruction via Orthogonal Neural Plane](https://mlsanity.com/paper/2312.15253/) (2023)
- [Semantic-SuPer: A Semantic-aware Surgical Perception Framework for Endoscopic Tissue Identification, Reconstruction, and Tracking](https://mlsanity.com/paper/2210.16674/) (2022)
- [Learning from Demonstrations for Autonomous Soft-tissue Retraction](https://mlsanity.com/paper/2110.00336/) (2021)
- [EndoSparse: Real-Time Sparse View Synthesis of Endoscopic Scenes using Gaussian Splatting](https://mlsanity.com/paper/2407.01029/) (2024)
- [Surgical Tattoos in Infrared: A Dataset for Quantifying Tissue Tracking and Mapping](https://mlsanity.com/paper/2309.16782/) (2023)
- [SENDD: Sparse Efficient Neural Depth and Deformation for Tissue Tracking](https://mlsanity.com/paper/2305.06477/) (2023)
- [SurgicalGS: Dynamic 3D Gaussian Splatting for Accurate Robotic-Assisted Surgical Scene Reconstruction](https://mlsanity.com/paper/2410.09292/) (2024)
- [SurgicalGaussian: Deformable 3D Gaussians for High-Fidelity Surgical Scene Reconstruction](https://mlsanity.com/paper/2407.05023/) (2024)
- [Image-based Pose Estimation and Shape Reconstruction for Robot Manipulators and Soft, Continuum Robots via Differentiable Rendering](https://mlsanity.com/paper/2302.14039/) (2023)
- [SculptBot: Pre-Trained Models for 3D Deformable Object Manipulation](https://mlsanity.com/paper/2309.08728/) (2023)
- [SurgRIPE challenge: Benchmark of Surgical Robot Instrument Pose Estimation](https://mlsanity.com/paper/2501.02990/) (2025)
- [Surgical Gym: A high-performance GPU-based platform for reinforcement learning with surgical robots](https://mlsanity.com/paper/2310.04676/) (2023)
- [Markerless Suture Needle 6D Pose Tracking with Robust Uncertainty Estimation for Autonomous Minimally Invasive Robotic Surgery](https://mlsanity.com/paper/2109.12722/) (2021)
- [RIDE: Self-Supervised Learning of Rotation-Equivariant Keypoint Detection and Invariant Description for Endoscopy](https://mlsanity.com/paper/2309.09563/) (2023)
- [Tracking Everything in Robotic-Assisted Surgery](https://mlsanity.com/paper/2409.19821/) (2024)
- [LiteTracker: Leveraging Temporal Causality for Accurate Low-latency Tissue Tracking](https://mlsanity.com/paper/2504.09904/) (2025)
- [Image Based Reconstruction of Liquids from 2D Surface Detections](https://mlsanity.com/paper/2111.11491/) (2021)
- [Robust Surgical Tool Tracking with Pixel-based Probabilities for Projected Geometric Primitives](https://mlsanity.com/paper/2403.04971/) (2024)
- [BodySLAM: A Generalized Monocular Visual SLAM Framework for Surgical Applications](https://mlsanity.com/paper/2408.03078/) (2024)
- [EndoGaussians: Single View Dynamic Gaussian Splatting for Deformable Endoscopic Tissues Reconstruction](https://mlsanity.com/paper/2401.13352/) (2024)
- [Ada-Tracker: Soft Tissue Tracking via Inter-Frame and Adaptive-Template Matching](https://mlsanity.com/paper/2403.06479/) (2024)

## Abstract (arXiv)

Traditional control and task automation have been successfully demonstrated in a variety of structured, controlled environments through the use of highly specialized modeled robotic systems in conjunction with multiple sensors. However, the application of autonomy in endoscopic surgery is very challenging, particularly in soft tissue work, due to the lack of high-quality images and the unpredictable, constantly deforming environment. In this work, we propose a novel surgical perception framework, SuPer, for surgical robotic control. This framework continuously collects 3D geometric information that allows for mapping a deformable surgical field while tracking rigid instruments within the field. To achieve this, a model-based tracker is employed to localize the surgical tool with a kinematic prior in conjunction with a model-free tracker to reconstruct the deformable environment and provide an estimated point cloud as a mapping of the environment. The proposed framework was implemented on the da Vinci Surgical System in real-time with an end-effector controller where the target configurations are set and regulated through the framework. Our proposed framework successfully completed soft tissue manipulation tasks with high accuracy. The demonstration of this novel framework is promising for the future of surgical autonomy. In addition, we provide our dataset for further surgical research.