# Realistic Surgical Image Dataset Generation Based On 3D Gaussian Splatting

arXiv: https://arxiv.org/abs/2407.14846 · HTML: https://mlsanity.com/paper/2407.14846/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 22 references that are in the map.
- First posted on arXiv 20 Jul 2024.

## Builds on (most cited first, 9 of 9)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023)
- [A Survey on 3D Gaussian Splatting](https://mlsanity.com/paper/2401.03890/) (2024)
- [MedGAN: Medical Image Translation using GANs](https://mlsanity.com/paper/1806.06397/) (2018)
- [Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery](https://mlsanity.com/paper/2206.15255/) (2022)
- [GANs for Medical Image Analysis](https://mlsanity.com/paper/1809.06222/) (2018)
- [EndoMapper dataset of complete calibrated endoscopy procedures](https://mlsanity.com/paper/2204.14240/) (2022)

## Built on by (most cited first, 6 of 6)

- [Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions](https://mlsanity.com/paper/2412.02621/) (2024)
- [Instrument-Splatting: Controllable Photorealistic Reconstruction of Surgical Instruments Using Gaussian Splatting](https://mlsanity.com/paper/2503.04082/) (2025)
- [Synthetic Dataset Generation for Autonomous Mobile Robots Using 3D Gaussian Splatting for Vision Training](https://mlsanity.com/paper/2506.05092/) (2025)
- [Implementing blind navigation through multi-modal sensing and gait guidance](https://mlsanity.com/paper/2506.19593/) (2025)
- [Surg-InvNeRF: Invertible NeRF for 3D tracking and reconstruction in surgical vision](https://mlsanity.com/paper/2508.09681/) (2025)
- [SurfSurg6D: Geometry Consistent Dense Correspondence for Textureless Surgical Instrument Pose Estimation](https://mlsanity.com/paper/2605.25598/) (2026)

## Abstract (arXiv)

Computer vision technologies markedly enhance the automation capabilities of robotic-assisted minimally invasive surgery (RAMIS) through advanced tool tracking, detection, and localization. However, the limited availability of comprehensive surgical datasets for training represents a significant challenge in this field. This research introduces a novel method that employs 3D Gaussian Splatting to generate synthetic surgical datasets. We propose a method for extracting and combining 3D Gaussian representations of surgical instruments and background operating environments, transforming and combining them to generate high-fidelity synthetic surgical scenarios. We developed a data recording system capable of acquiring images alongside tool and camera poses in a surgical scene. Using this pose data, we synthetically replicate the scene, thereby enabling direct comparisons of the synthetic image quality (29.592 PSNR). As a further validation, we compared two YOLOv5 models trained on the synthetic and real data, respectively, and assessed their performance in an unseen real-world test dataset. Comparing the performances, we observe an improvement in neural network performance, with the synthetic-trained model outperforming the real-world trained model by 12%, testing both on real-world data.