# Real-to-Sim Registration of Deformable Soft Tissue with Position-Based Dynamics for Surgical Robot Autonomy

arXiv: https://arxiv.org/abs/2011.00800 · HTML: https://mlsanity.com/paper/2011.00800/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Nov 2020.

## Built on by (most cited first, 11 of 11)

- [DiffCloud: Real-to-Sim from Point Clouds with Differentiable Simulation and Rendering of Deformable Objects](https://mlsanity.com/paper/2204.03139/) (2022)
- [A Bayesian Treatment of Real-to-Sim for Deformable Object Manipulation](https://mlsanity.com/paper/2112.05068/) (2021)
- [SculptBot: Pre-Trained Models for 3D Deformable Object Manipulation](https://mlsanity.com/paper/2309.08728/) (2023)
- [SculptDiff: Learning Robotic Clay Sculpting from Humans with Goal Conditioned Diffusion Policy](https://mlsanity.com/paper/2403.10401/) (2024)
- [BASED: Bundle-Adjusting Surgical Endoscopic Dynamic Video Reconstruction using Neural Radiance Fields](https://mlsanity.com/paper/2309.15329/) (2023)
- [A Review on Organ Deformation Modeling Approaches for Reliable Surgical Navigation using Augmented Reality](https://mlsanity.com/paper/2408.02713/) (2024)
- [Robust Surgical Tool Tracking with Pixel-based Probabilities for Projected Geometric Primitives](https://mlsanity.com/paper/2403.04971/) (2024)
- [ORRN: An ODE-based Recursive Registration Network for Deformable Respiratory Motion Estimation with Lung 4DCT Images](https://mlsanity.com/paper/2305.14673/) (2023)
- [SuPerPM: A Surgical Perception Framework Based on Deep Point Matching Learned from Physical Constrained Simulation Data](https://mlsanity.com/paper/2309.13863/) (2023)
- [arg-VU: Affordance Reasoning with Physics-Aware 3D Geometry for Visual Understanding in Robotic Surgery](https://mlsanity.com/paper/2603.26814/) (2026)
- [RealSimLoop: Online Real-to-Sim Adaptation via Differentiable Reduced-Order Simulation with Vision Feedback](https://mlsanity.com/paper/2609.09828/) (2026)

## Abstract (arXiv)

Autonomy in robotic surgery is very challenging in unstructured environments, especially when interacting with deformable soft tissues. The main difficulty is to generate model-based control methods that account for deformation dynamics during tissue manipulation. Previous works in vision-based perception can capture the geometric changes within the scene, however, model-based controllers integrated with dynamic properties, a more accurate and safe approach, has not been studied before. Considering the mechanic coupling between the robot and the environment, it is crucial to develop a registered, simulated dynamical model. In this work, we propose an online, continuous, real-to-sim registration method to bridge 3D visual perception with position-based dynamics (PBD) modeling of tissues. The PBD method is employed to simulate soft tissue dynamics as well as rigid tool interactions for model-based control. Meanwhile, a vision-based strategy is used to generate 3D reconstructed point cloud surfaces based on real-world manipulation, so as to register and update the simulation. To verify this real-to-sim approach, tissue experiments have been conducted on the da Vinci Research Kit. Our real-to-sim approach successfully reduces registration error online, which is especially important for safety during autonomous control. Moreover, it achieves higher accuracy in occluded areas than fusion-based reconstruction.