# EndoGaussian: Real-time Gaussian Splatting for Dynamic Endoscopic Scene Reconstruction

arXiv: https://arxiv.org/abs/2401.12561 · HTML: https://mlsanity.com/paper/2401.12561/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 23 Jan 2024.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [HexPlane: A Fast Representation for Dynamic Scenes](https://mlsanity.com/paper/2301.09632/) (2023)
- [FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting](https://mlsanity.com/paper/2312.00451/) (2023)
- [A Survey on 3D Gaussian Splatting](https://mlsanity.com/paper/2401.03890/) (2024)
- [Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2011.02910/) (2020)
- [Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery](https://mlsanity.com/paper/2206.15255/) (2022)
- [Stereo Correspondence and Reconstruction of Endoscopic Data Challenge](https://mlsanity.com/paper/2101.01133/) (2021)
- [EndoSurf: Neural Surface Reconstruction of Deformable Tissues with Stereo Endoscope Videos](https://mlsanity.com/paper/2307.11307/) (2023)
- [SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation with Surgical Robotics](https://mlsanity.com/paper/1909.05405/) (2019)
- [E-DSSR: Efficient Dynamic Surgical Scene Reconstruction with Transformer-based Stereoscopic Depth Perception](https://mlsanity.com/paper/2107.00229/) (2021)
- [Neural LerPlane Representations for Fast 4D Reconstruction of Deformable Tissues](https://mlsanity.com/paper/2305.19906/) (2023)
- [SurfelWarp: Efficient Non-Volumetric Single View Dynamic Reconstruction](https://mlsanity.com/paper/1904.13073/) (2019)
- [LightNeuS: Neural Surface Reconstruction in Endoscopy using Illumination Decline](https://mlsanity.com/paper/2309.02777/) (2023)
- [Reconstructing Sinus Anatomy from Endoscopic Video -- Towards a Radiation-free Approach for Quantitative Longitudinal Assessment](https://mlsanity.com/paper/2003.08502/) (2020)
- [Dynamic Reconstruction of Deformable Soft-tissue with Stereo Scope in Minimal Invasive Surgery](https://mlsanity.com/paper/2003.10867/) (2020)
- [Real-time Dense Reconstruction of Tissue Surface from Stereo Optical Video](https://mlsanity.com/paper/2007.12623/) (2020)

## Built on by (most cited first, 22 of 22)

- [Multi-Layer Gaussian Splatting for Immersive Anatomy Visualization](https://mlsanity.com/paper/2410.16978/) (2024)
- [EndoWave: 4D Gaussian Splatting with Rational Wavelet for Endoscopic Reconstruction](https://mlsanity.com/paper/2510.23087/) (2025)
- [SimEndoGS: Efficient Data-driven Scene Simulation using Robotic Surgery Videos via Physics-embedded 3D Gaussians](https://mlsanity.com/paper/2405.00956/) (2024)
- [HEROS-GAN: Honed-Energy Regularized and Optimal Supervised GAN for Enhancing Accuracy and Range of Low-Cost Accelerometers](https://mlsanity.com/paper/2502.18064/) (2025)
- [NeRFscopy: Neural Radiance Fields for in-vivo Time-Varying Tissues from Endoscopy](https://mlsanity.com/paper/2602.15775/) (2026)
- [Diff2DGS: Reliable Reconstruction of Occluded Surgical Scenes via 2D Gaussian Splatting](https://mlsanity.com/paper/2602.18314/) (2026)
- [ColonSplat: Reconstruction of Peristaltic Motion in Colonoscopy with Dynamic Gaussian Splatting](https://mlsanity.com/paper/2603.06860/) (2026)
- [EndoLRMGS: Complete Endoscopic Scene Reconstruction combining Large Reconstruction Modelling and Gaussian Splatting](https://mlsanity.com/paper/2503.22437/) (2025)
- [Endoscopic Depth Estimation Based on Deep Learning: A Survey](https://mlsanity.com/paper/2507.20881/) (2025)
- [Feature-EndoGaussian: Feature Distilled Gaussian Splatting in Surgical Deformable Scene Reconstruction](https://mlsanity.com/paper/2503.06161/) (2025)
- [NeeCo: Image Synthesis of Novel Instrument States Based on Dynamic and Deformable 3D Gaussian Reconstruction](https://mlsanity.com/paper/2508.07897/) (2025)
- [Efficient 3D Scene Reconstruction and Simulation from Sparse Endoscopic Views](https://mlsanity.com/paper/2509.17027/) (2025)
- [SuperQuadricOcc: Real-Time Self-Supervised Semantic Occupancy Estimation with Superquadric Volume Rendering](https://mlsanity.com/paper/2511.17361/) (2025)
- [G-SHARP: Gaussian Surgical Hardware Accelerated Real-time Pipeline](https://mlsanity.com/paper/2512.02482/) (2025)
- [Monocular Endoscopic Tissue 3D Reconstruction with Multi-Level Geometry Regularization](https://mlsanity.com/paper/2602.20718/) (2026)
- [The Dresden Dataset for 4D Reconstruction of Non-Rigid Abdominal Surgical Scenes](https://mlsanity.com/paper/2603.02985/) (2026)
- [Investigating a Policy-Based Formulation for Endoscopic Camera Pose Recovery](https://mlsanity.com/paper/2603.20045/) (2026)
- [EndoVGGT: GNN-Enhanced Depth Estimation for Surgical 3D Reconstruction](https://mlsanity.com/paper/2603.24577/) (2026)
- [Gastroendoscopy View Synthesis: A New Real Dataset and Evaluation](https://mlsanity.com/paper/2606.25427/) (2026)
- [DeGenseGS: Geometrically and Semantically Decoupled Surgical Scene Understanding in 4D Gaussian Splatting](https://mlsanity.com/paper/2607.04761/) (2026)
- [SCOPE-4D: Endoscopic 4D Geometry Foundation Models](https://mlsanity.com/paper/2610.02343/) (2026)
- [Hierarchical Curriculum Learning for Surgical Video Generation: Focusing on Bilateral Axillo-Breast Approach Robotic Thyroidectomy Datasets](https://mlsanity.com/paper/2610.17684/) (2026)

## Abstract (arXiv)

Reconstructing deformable tissues from endoscopic videos is essential in many downstream surgical applications. However, existing methods suffer from slow rendering speed, greatly limiting their practical use. In this paper, we introduce EndoGaussian, a real-time endoscopic scene reconstruction framework built on 3D Gaussian Splatting (3DGS). By integrating the efficient Gaussian representation and highly-optimized rendering engine, our framework significantly boosts the rendering speed to a real-time level. To adapt 3DGS for endoscopic scenes, we propose two strategies, Holistic Gaussian Initialization (HGI) and Spatio-temporal Gaussian Tracking (SGT), to handle the non-trivial Gaussian initialization and tissue deformation problems, respectively. In HGI, we leverage recent depth estimation models to predict depth maps of input binocular/monocular image sequences, based on which pixels are re-projected and combined for holistic initialization. In SPT, we propose to model surface dynamics using a deformation field, which is composed of an efficient encoding voxel and a lightweight deformation decoder, allowing for Gaussian tracking with minor training and rendering burden. Experiments on public datasets demonstrate our efficacy against prior SOTAs in many aspects, including better rendering speed (195 FPS real-time, 100× gain), better rendering quality (37.848 PSNR), and less training overhead (within 2 min/scene), showing significant promise for intraoperative surgery applications. Code is available at: urlhttps://yifliu3.github.io/EndoGaussian/.