# Toward Reliable AR-Guided Surgical Navigation: Interactive Deformation Modeling with Data-Driven Biomechanics and Prompts

arXiv: https://arxiv.org/abs/2506.08048 · HTML: https://mlsanity.com/paper/2506.08048/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 33 references that are in the map.
- First posted on arXiv 8 Jun 2025.

## Builds on (most cited first, 14 of 14)

- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Non-rigid Point Cloud Registration with Neural Deformation Pyramid](https://mlsanity.com/paper/2205.12796/) (2022)
- [Learning Soft Tissue Behavior of Organs for Surgical Navigation with Convolutional Neural Networks](https://mlsanity.com/paper/1904.00722/) (2019)
- [Non-Rigid Volume to Surface Registration using a Data-Driven Biomechanical Model](https://mlsanity.com/paper/2005.14695/) (2020)
- [Deep Graph-based Spatial Consistency for Robust Non-rigid Point Cloud Registration](https://mlsanity.com/paper/2303.09950/) (2023)
- [Boundary Constraint-free Biomechanical Model-Based Surface Matching for Intraoperative Liver Deformation Correction](https://mlsanity.com/paper/2403.09964/) (2024)
- [Utility of Optical See-Through Head Mounted Displays in Augmented Reality-Assisted Surgery: A systematic review](https://mlsanity.com/paper/2202.04141/) (2022)
- [Non-rigid Medical Image Registration using Physics-informed Neural Networks](https://mlsanity.com/paper/2302.10343/) (2023)
- [Landmark-Free Preoperative-to-Intraoperative Registration in Laparoscopic Liver Resection](https://mlsanity.com/paper/2504.15152/) (2025)
- [UTOPIC: Uncertainty-aware Overlap Prediction Network for Partial Point Cloud Registration](https://mlsanity.com/paper/2208.02712/) (2022)
- [An Optimal Control Problem for Elastic Registration and Force Estimation in Augmented Surgery](https://mlsanity.com/paper/2206.10931/) (2022)
- [A Review on Organ Deformation Modeling Approaches for Reliable Surgical Navigation using Augmented Reality](https://mlsanity.com/paper/2408.02713/) (2024)
- [Comparing Regularized Kelvinlet Functions and the Finite Element Method for Registration of Medical Images to Sparse Organ Data](https://mlsanity.com/paper/2306.05509/) (2023)
- [Tracking Tumors under Deformation from Partial Point Clouds using Occupancy Networks](https://mlsanity.com/paper/2411.02619/) (2024)

## Built on by (most cited first, 3 of 3)

- [PIVOTS: Aligning unseen Structures using Preoperative to Intraoperative Volume-To-Surface Registration for Liver Navigation](https://mlsanity.com/paper/2507.20337/) (2025)
- [SurgiATM: A Physics-Guided Plug-and-Play Model for Deep Learning-Based Smoke Removal in Laparoscopic Surgery](https://mlsanity.com/paper/2511.05059/) (2025)
- [ARport: An Augmented Reality System for Markerless Image-Guided Port Placement in Robotic Surgery](https://mlsanity.com/paper/2602.14153/) (2026)

## Abstract (arXiv)

In augmented reality (AR)-guided surgical navigation, preoperative organ models are superimposed onto the patient's intraoperative anatomy to visualize critical structures such as vessels and tumors. Accurate deformation modeling is essential to maintain the reliability of AR overlays by ensuring alignment between preoperative models and the dynamically changing anatomy. Although the finite element method (FEM) offers physically plausible modeling, its high computational cost limits intraoperative applicability. Moreover, existing algorithms often fail to handle large anatomical changes, such as those induced by pneumoperitoneum or ligament dissection, leading to inaccurate anatomical correspondences and compromised AR guidance. To address these challenges, we propose a data-driven biomechanics algorithm that preserves FEM-level accuracy while improving computational efficiency. In addition, we introduce a novel human-in-the-loop mechanism into the deformation modeling process. This enables surgeons to interactively provide prompts to correct anatomical misalignments, thereby incorporating clinical expertise and allowing the model to adapt dynamically to complex surgical scenarios. Experiments on a publicly available dataset demonstrate that our algorithm achieves a mean target registration error of 3.42 mm. Incorporating surgeon prompts through the interactive framework further reduces the error to 2.78 mm, surpassing state-of-the-art methods in volumetric accuracy. These results highlight the ability of our framework to deliver efficient and accurate deformation modeling while enhancing surgeon-algorithm collaboration, paving the way for safer and more reliable computer-assisted surgeries.