# Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting

arXiv: https://arxiv.org/abs/2401.16416 · HTML: https://mlsanity.com/paper/2401.16416/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 29 Jan 2024.

## Builds on (most cited first, 24 of 24)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [HexPlane: A Fast Representation for Dynamic Scenes](https://mlsanity.com/paper/2301.09632/) (2023)
- [Fast Dynamic Radiance Fields with Time-Aware Neural Voxels](https://mlsanity.com/paper/2205.15285/) (2022)
- [SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis](https://mlsanity.com/paper/2303.16196/) (2023)
- [Differentiable Surface Splatting for Point-based Geometry Processing](https://mlsanity.com/paper/1906.04173/) (2019)
- [A Survey on 3D Gaussian Splatting](https://mlsanity.com/paper/2401.03890/) (2024)
- [Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2011.02910/) (2020)
- [Depth-Regularized Optimization for 3D Gaussian Splatting in Few-Shot Images](https://mlsanity.com/paper/2311.13398/) (2023)
- [Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery](https://mlsanity.com/paper/2206.15255/) (2022)
- [Stereo Correspondence and Reconstruction of Endoscopic Data Challenge](https://mlsanity.com/paper/2101.01133/) (2021)
- [Self-Supervised Monocular Depth and Ego-Motion Estimation in Endoscopy: Appearance Flow to the Rescue](https://mlsanity.com/paper/2112.08122/) (2021)
- [EndoSurf: Neural Surface Reconstruction of Deformable Tissues with Stereo Endoscope Videos](https://mlsanity.com/paper/2307.11307/) (2023)
- [SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challenge](https://mlsanity.com/paper/2401.00496/) (2023)
- [Surgical-DINO: Adapter Learning of Foundation Models for Depth Estimation in Endoscopic Surgery](https://mlsanity.com/paper/2401.06013/) (2024)
- [Learning How To Robustly Estimate Camera Pose in Endoscopic Videos](https://mlsanity.com/paper/2304.08023/) (2023)
- [E-DSSR: Efficient Dynamic Surgical Scene Reconstruction with Transformer-based Stereoscopic Depth Perception](https://mlsanity.com/paper/2107.00229/) (2021)
- [Neural LerPlane Representations for Fast 4D Reconstruction of Deformable Tissues](https://mlsanity.com/paper/2305.19906/) (2023)
- [Masked Space-Time Hash Encoding for Efficient Dynamic Scene Reconstruction](https://mlsanity.com/paper/2310.17527/) (2023)
- [EndoGS: Deformable Endoscopic Tissues Reconstruction with Gaussian Splatting](https://mlsanity.com/paper/2401.11535/) (2024)
- [ORBSLAM-based Endoscope Tracking and 3D Reconstruction](https://mlsanity.com/paper/1608.08149/) (2016)
- [Intuitive Surgical SurgToolLoc and SurgVU Challenges Results: 2022-2025](https://mlsanity.com/paper/2305.07152/) (2023)
- [ResDepth: Learned Residual Stereo Reconstruction](https://mlsanity.com/paper/2001.08026/) (2020)

## Built on by (most cited first, 30 of 43)

- [Recent Advances in 3D Gaussian Splatting](https://mlsanity.com/paper/2403.11134/) (2024)
- [Advancing Dense Endoscopic Reconstruction with Gaussian Splatting-driven Surface Normal-aware Tracking and Mapping](https://mlsanity.com/paper/2501.19319/) (2025)
- [MonoPCC: Photometric-invariant Cycle Constraint for Monocular Depth Estimation of Endoscopic Images](https://mlsanity.com/paper/2404.16571/) (2024)
- [Digital Twin Generation from Visual Data: A Survey](https://mlsanity.com/paper/2504.13159/) (2025)
- [SurgTPGS: Semantic 3D Surgical Scene Understanding with Text Promptable Gaussian Splatting](https://mlsanity.com/paper/2506.23309/) (2025)
- [PMGS: Reconstruction of Projectile Motion Across Large Spatiotemporal Spans via 3D Gaussian Splatting](https://mlsanity.com/paper/2508.02660/) (2025)
- [PR-ENDO: Physically Based Relightable Gaussian Splatting for Endoscopy](https://mlsanity.com/paper/2411.12510/) (2024)
- [NFL-BA: Near-Field Light Bundle Adjustment for SLAM in Dynamic Lighting](https://mlsanity.com/paper/2412.13176/) (2024)
- [BridgeSplat: Bidirectionally Coupled CT and Non-Rigid Gaussian Splatting for Deformable Intraoperative Surgical Navigation](https://mlsanity.com/paper/2509.18501/) (2025)
- [EndoWave: 4D Gaussian Splatting with Rational Wavelet for Endoscopic Reconstruction](https://mlsanity.com/paper/2510.23087/) (2025)
- [SimEndoGS: Efficient Data-driven Scene Simulation using Robotic Surgery Videos via Physics-embedded 3D Gaussians](https://mlsanity.com/paper/2405.00956/) (2024)
- [Registering Neural 4D Gaussians for Endoscopic Surgery](https://mlsanity.com/paper/2407.20213/) (2024)
- [Real-Time Spatio-Temporal Reconstruction of Dynamic Endoscopic Scenes with 4D Gaussian Splatting](https://mlsanity.com/paper/2411.01218/) (2024)
- [Realistic Surgical Simulation from Monocular Videos](https://mlsanity.com/paper/2412.02359/) (2024)
- [Endo-4DGX: Robust Endoscopic Scene Reconstruction and Illumination Correction with Gaussian Splatting](https://mlsanity.com/paper/2506.23308/) (2025)
- [ColonSplat: Reconstruction of Peristaltic Motion in Colonoscopy with Dynamic Gaussian Splatting](https://mlsanity.com/paper/2603.06860/) (2026)
- [EndoLRMGS: Complete Endoscopic Scene Reconstruction combining Large Reconstruction Modelling and Gaussian Splatting](https://mlsanity.com/paper/2503.22437/) (2025)
- [TR2M: Transferring Monocular Relative Depth to Metric Depth with Language Descriptions and Dual-Level Scale-Oriented Contrast](https://mlsanity.com/paper/2506.13387/) (2025)
- [Moving Light Adaptive Colonoscopy Reconstruction via Illumination-Attenuation-Aware 3D Gaussian Splatting](https://mlsanity.com/paper/2510.18739/) (2025)
- [More than Segmentation: Benchmarking SAM 3 for Segmentation, 3D Perception, and Reconstruction in Robotic Surgery](https://mlsanity.com/paper/2512.07596/) (2025)
- [4D Monocular Surgical Reconstruction under Arbitrary Camera Motions](https://mlsanity.com/paper/2602.17473/) (2026)
- [Stop Holding Your Breath: CT-Informed Gaussian Splatting for Dynamic Bronchoscopy](https://mlsanity.com/paper/2604.28179/) (2026)
- [EndoGSim: Physics-Aware 4D Dynamic Endoscopic Scene Simulations via MLLM-Guided Gaussian Splatting](https://mlsanity.com/paper/2605.16022/) (2026)
- [Parameter-Efficient Adaptation of SAM3 for Prompt-Driven Surgical Concept Segmentation](https://mlsanity.com/paper/2607.23694/) (2026)
- [Head Pose Estimation and 3D Neural Surface Reconstruction via Monocular Camera in situ for Navigation and Safe Insertion into Natural Openings](https://mlsanity.com/paper/2406.13048/) (2024)
- [V2-SfMLearner: Learning Monocular Depth and Ego-motion for Multimodal Wireless Capsule Endoscopy](https://mlsanity.com/paper/2412.17595/) (2024)
- [Tracking-Aware Deformation Field Estimation for Non-rigid 3D Reconstruction in Robotic Surgeries](https://mlsanity.com/paper/2503.02558/) (2025)
- [Can DeepSeek Reason Like a Surgeon? An Empirical Evaluation for Vision-Language Understanding in Robotic-Assisted Surgery](https://mlsanity.com/paper/2503.23130/) (2025)
- [NeeCo: Image Synthesis of Novel Instrument States Based on Dynamic and Deformable 3D Gaussian Reconstruction](https://mlsanity.com/paper/2508.07897/) (2025)
- [Efficient 3D Scene Reconstruction and Simulation from Sparse Endoscopic Views](https://mlsanity.com/paper/2509.17027/) (2025)

## Abstract (arXiv)

In the realm of robot-assisted minimally invasive surgery, dynamic scene reconstruction can significantly enhance downstream tasks and improve surgical outcomes. Neural Radiance Fields (NeRF)-based methods have recently risen to prominence for their exceptional ability to reconstruct scenes but are hampered by slow inference speed, prolonged training, and inconsistent depth estimation. Some previous work utilizes ground truth depth for optimization but is hard to acquire in the surgical domain. To overcome these obstacles, we present Endo-4DGS, a real-time endoscopic dynamic reconstruction approach that utilizes 3D Gaussian Splatting (GS) for 3D representation. Specifically, we propose lightweight MLPs to capture temporal dynamics with Gaussian deformation fields. To obtain a satisfactory Gaussian Initialization, we exploit a powerful depth estimation foundation model, Depth-Anything, to generate pseudo-depth maps as a geometry prior. We additionally propose confidence-guided learning to tackle the ill-pose problems in monocular depth estimation and enhance the depth-guided reconstruction with surface normal constraints and depth regularization. Our approach has been validated on two surgical datasets, where it can effectively render in real-time, compute efficiently, and reconstruct with remarkable accuracy.