# SuPerPM: A Surgical Perception Framework Based on Deep Point Matching Learned from Physical Constrained Simulation Data

arXiv: https://arxiv.org/abs/2309.13863 · HTML: https://mlsanity.com/paper/2309.13863/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 44 references that are in the map.
- First posted on arXiv 25 Sep 2023.

## Builds on (most cited first, 26 of 26)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching](https://mlsanity.com/paper/2109.07547/) (2021)
- [Vector Neurons: A General Framework for SO(3)-Equivariant Networks](https://mlsanity.com/paper/2104.12229/) (2021)
- [Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation](https://mlsanity.com/paper/2112.05124/) (2021)
- [3D-CODED : 3D Correspondences by Deep Deformation](https://mlsanity.com/paper/1806.05228/) (2018)
- [4DComplete: Non-Rigid Motion Estimation Beyond the Observable Surface](https://mlsanity.com/paper/2105.01905/) (2021)
- [Robust Point Cloud Registration Framework Based on Deep Graph Matching](https://mlsanity.com/paper/2103.04256/) (2021)
- [Lepard: Learning partial point cloud matching in rigid and deformable scenes](https://mlsanity.com/paper/2111.12591/) (2021)
- [Endo-Depth-and-Motion: Reconstruction and Tracking in Endoscopic Videos using Depth Networks and Photometric Constraints](https://mlsanity.com/paper/2103.16525/) (2021)
- [DeepDeform: Learning Non-rigid RGB-D Reconstruction with Semi-supervised Data](https://mlsanity.com/paper/1912.04302/) (2019)
- [VolumeDeform: Real-time Volumetric Non-rigid Reconstruction](https://mlsanity.com/paper/1603.08161/) (2016)
- [Neural Non-Rigid Tracking](https://mlsanity.com/paper/2006.13240/) (2020)
- [SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation with Surgical Robotics](https://mlsanity.com/paper/1909.05405/) (2019)
- [DefSLAM: Tracking and Mapping of Deforming Scenes from Monocular Sequences](https://mlsanity.com/paper/1908.08918/) (2019)
- [SurfelWarp: Efficient Non-Volumetric Single View Dynamic Reconstruction](https://mlsanity.com/paper/1904.13073/) (2019)
- [SuPer Deep: A Surgical Perception Framework for Robotic Tissue Manipulation using Deep Learning for Feature Extraction](https://mlsanity.com/paper/2003.03472/) (2020)
- [SD-DefSLAM: Semi-Direct Monocular SLAM for Deformable and Intracorporeal Scenes](https://mlsanity.com/paper/2010.09409/) (2020)
- [Semantic-SuPer: A Semantic-aware Surgical Perception Framework for Endoscopic Tissue Identification, Reconstruction, and Tracking](https://mlsanity.com/paper/2210.16674/) (2022)
- [Robot Autonomy for Surgery](https://mlsanity.com/paper/1707.03080/) (2017)
- [Robotic Tool Tracking under Partially Visible Kinematic Chain: A Unified Approach](https://mlsanity.com/paper/2102.06235/) (2021)
- [Learning to Optimize Non-Rigid Tracking](https://mlsanity.com/paper/2003.12230/) (2020)
- [Real-to-Sim Registration of Deformable Soft Tissue with Position-Based Dynamics for Surgical Robot Autonomy](https://mlsanity.com/paper/2011.00800/) (2020)
- [Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Clouds](https://mlsanity.com/paper/2110.04685/) (2021)
- [Polyp Detection and Segmentation from Video Capsule Endoscopy: A Review](https://mlsanity.com/paper/1609.01915/) (2016)
- [From Bench to Bedside: The First Live Robotic Surgery on the dVRK to Enable Remote Telesurgery with Motion Scaling](https://mlsanity.com/paper/2109.12177/) (2021)

## Built on by (most cited first, 2 of 2)

- [Segment Anything for Videos: A Systematic Survey](https://mlsanity.com/paper/2408.08315/) (2024)
- [arg-VU: Affordance Reasoning with Physics-Aware 3D Geometry for Visual Understanding in Robotic Surgery](https://mlsanity.com/paper/2603.26814/) (2026)

## Abstract (arXiv)

A major source of endoscopic tissue tracking errors during deformations stems from wrong data association between observed sensor measurements with previously tracked scene. To mitigate this issue, we present a surgical perception framework, SuPerPM, that leverages learning-based non-rigid point cloud matching for data association, thus accommodating larger deformations than previous approaches which relied on Iterative Closest Point (ICP) for point associations. The learning models typically require training data with ground truth point cloud correspondences, which is challenging or even impractical to collect in surgical environments. Thus, for tuning the learning model, we gather endoscopic data of soft tissue being manipulated by a surgical robot and then establish correspondences between point clouds at different time points to serve as ground truth. This was achieved by employing a position-based dynamics (PBD) simulation to ensure that the correspondences adhered to physical constraints. The proposed framework is demonstrated on several challenging surgical datasets that are characterized by large deformations, achieving superior performance over advanced surgical scene tracking algorithms.