# Virtual Reality for Synergistic Surgical Training and Data Generation

arXiv: https://arxiv.org/abs/2111.08097 · HTML: https://mlsanity.com/paper/2111.08097/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2021.

## Built on by (most cited first, 6 of 6)

- [Human-in-the-loop Embodied Intelligence with Interactive Simulation Environment for Surgical Robot Learning](https://mlsanity.com/paper/2301.00452/) (2023)
- [Pelphix: Surgical Phase Recognition from X-ray Images in Percutaneous Pelvic Fixation](https://mlsanity.com/paper/2304.09285/) (2023)
- [TAToo: Vision-based Joint Tracking of Anatomy and Tool for Skull-base Surgery](https://mlsanity.com/paper/2212.14131/) (2022)
- [LiteVR: Interpretable and Lightweight Cybersickness Detection using Explainable AI](https://mlsanity.com/paper/2302.03037/) (2023)
- [TwinOR: Photorealistic Digital Twins of Dynamic Operating Rooms for Embodied AI Research](https://mlsanity.com/paper/2511.07412/) (2025)
- [Twin-S: A Digital Twin for Skull-base Surgery](https://mlsanity.com/paper/2211.11863/) (2022)

## Abstract (arXiv)

Surgical simulators not only allow planning and training of complex procedures, but also offer the ability to generate structured data for algorithm development, which may be applied in image-guided computer assisted interventions. While there have been efforts on either developing training platforms for surgeons or data generation engines, these two features, to our knowledge, have not been offered together. We present our developments of a cost-effective and synergistic framework, named Asynchronous Multibody Framework Plus (AMBF+), which generates data for downstream algorithm development simultaneously with users practicing their surgical skills. AMBF+ offers stereoscopic display on a virtual reality (VR) device and haptic feedback for immersive surgical simulation. It can also generate diverse data such as object poses and segmentation maps. AMBF+ is designed with a flexible plugin setup which allows for unobtrusive extension for simulation of different surgical procedures. We show one use case of AMBF+ as a virtual drilling simulator for lateral skull-base surgery, where users can actively modify the patient anatomy using a virtual surgical drill. We further demonstrate how the data generated can be used for validating and training downstream computer vision algorithms