# EndoSparse: Real-Time Sparse View Synthesis of Endoscopic Scenes using Gaussian Splatting

arXiv: https://arxiv.org/abs/2407.01029 · HTML: https://mlsanity.com/paper/2407.01029/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 68 references that are in the map.
- First posted on arXiv 1 Jul 2024.

## Builds on (most cited first, 30 of 44)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting](https://mlsanity.com/paper/2312.00451/) (2023)
- [U-KAN Makes Strong Backbone for Medical Image Segmentation and Generation](https://mlsanity.com/paper/2406.02918/) (2024)
- [Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners](https://mlsanity.com/paper/2303.02151/) (2023)
- [Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery](https://mlsanity.com/paper/2206.15255/) (2022)
- [Stereo Correspondence and Reconstruction of Endoscopic Data Challenge](https://mlsanity.com/paper/2101.01133/) (2021)
- [CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention](https://mlsanity.com/paper/2209.14169/) (2022)
- [SIGMA: Semantic-complete Graph Matching for Domain Adaptive Object Detection](https://mlsanity.com/paper/2203.06398/) (2022)
- [3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentation](https://mlsanity.com/paper/2306.13465/) (2023)
- [EndoSurf: Neural Surface Reconstruction of Deformable Tissues with Stereo Endoscope Videos](https://mlsanity.com/paper/2307.11307/) (2023)
- [Endora: Video Generation Models as Endoscopy Simulators](https://mlsanity.com/paper/2403.11050/) (2024)
- [Surgical-DINO: Adapter Learning of Foundation Models for Depth Estimation in Endoscopic Surgery](https://mlsanity.com/paper/2401.06013/) (2024)
- [Few-shot Medical Image Segmentation using a Global Correlation Network with Discriminative Embedding](https://mlsanity.com/paper/2012.05440/) (2020)
- [SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation with Surgical Robotics](https://mlsanity.com/paper/1909.05405/) (2019)
- [E-DSSR: Efficient Dynamic Surgical Scene Reconstruction with Transformer-based Stereoscopic Depth Perception](https://mlsanity.com/paper/2107.00229/) (2021)
- [Neural LerPlane Representations for Fast 4D Reconstruction of Deformable Tissues](https://mlsanity.com/paper/2305.19906/) (2023)
- [Domain Generalization on Medical Imaging Classification using Episodic Training with Task Augmentation](https://mlsanity.com/paper/2106.06908/) (2021)
- [EndoGS: Deformable Endoscopic Tissues Reconstruction with Gaussian Splatting](https://mlsanity.com/paper/2401.11535/) (2024)
- [StegaNeRF: Embedding Invisible Information within Neural Radiance Fields](https://mlsanity.com/paper/2212.01602/) (2022)
- [3DGR-CT: Sparse-View CT Reconstruction with a 3D Gaussian Representation](https://mlsanity.com/paper/2312.15676/) (2023)

## Built on by (most cited first, 16 of 16)

- [Large Spatial Model: End-to-end Unposed Images to Semantic 3D](https://mlsanity.com/paper/2410.18956/) (2024)
- [The NeRF Signature: Codebook-Aided Watermarking for Neural Radiance Fields](https://mlsanity.com/paper/2502.19125/) (2025)
- [GTP-4o: Modality-prompted Heterogeneous Graph Learning for Omni-modal Biomedical Representation](https://mlsanity.com/paper/2407.05540/) (2024)
- [A Review of 3D Reconstruction Techniques for Deformable Tissues in Robotic Surgery](https://mlsanity.com/paper/2408.04426/) (2024)
- [Deformable Gaussian Splatting for Efficient and High-Fidelity Reconstruction of Surgical Scenes](https://mlsanity.com/paper/2501.01101/) (2025)
- [When 3D Partial Points Meets SAM: Tooth Point Cloud Segmentation with Sparse Labels](https://mlsanity.com/paper/2409.01691/) (2024)
- [PR-ENDO: Physically Based Relightable Gaussian Splatting for Endoscopy](https://mlsanity.com/paper/2411.12510/) (2024)
- [BridgeSplat: Bidirectionally Coupled CT and Non-Rigid Gaussian Splatting for Deformable Intraoperative Surgical Navigation](https://mlsanity.com/paper/2509.18501/) (2025)
- [From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generation](https://mlsanity.com/paper/2605.08712/) (2026)
- [ConcealGS: Concealing Invisible Copyright Information in 3D Gaussian Splatting](https://mlsanity.com/paper/2501.03605/) (2025)
- [ColonSplat: Reconstruction of Peristaltic Motion in Colonoscopy with Dynamic Gaussian Splatting](https://mlsanity.com/paper/2603.06860/) (2026)
- [RDG-GS: Relative Depth Guidance with Gaussian Splatting for Real-time Sparse-View 3D Rendering](https://mlsanity.com/paper/2501.11102/) (2025)
- [Moving Light Adaptive Colonoscopy Reconstruction via Illumination-Attenuation-Aware 3D Gaussian Splatting](https://mlsanity.com/paper/2510.18739/) (2025)
- [Hybrid Foveated Path Tracing with Peripheral Gaussians for Immersive Anatomy](https://mlsanity.com/paper/2601.22026/) (2026)
- [4D Monocular Surgical Reconstruction under Arbitrary Camera Motions](https://mlsanity.com/paper/2602.17473/) (2026)
- [Monocular Endoscopic Tissue 3D Reconstruction with Multi-Level Geometry Regularization](https://mlsanity.com/paper/2602.20718/) (2026)

## Abstract (arXiv)

3D reconstruction of biological tissues from a collection of endoscopic images is a key to unlock various important downstream surgical applications with 3D capabilities. Existing methods employ various advanced neural rendering techniques for photorealistic view synthesis, but they often struggle to recover accurate 3D representations when only sparse observations are available, which is usually the case in real-world clinical scenarios. To tackle this sparsity challenge, we propose a framework leveraging the prior knowledge from multiple foundation models during the reconstruction process, dubbed as textitEndoSparse. Experimental results indicate that our proposed strategy significantly improves the geometric and appearance quality under challenging sparse-view conditions, including using only three views. In rigorous benchmarking experiments against state-of-the-art methods, textitEndoSparse achieves superior results in terms of accurate geometry, realistic appearance, and rendering efficiency, confirming the robustness to sparse-view limitations in endoscopic reconstruction. textitEndoSparse signifies a steady step towards the practical deployment of neural 3D reconstruction in real-world clinical scenarios. Project page: https://endo-sparse.github.io/.