# Neural LerPlane Representations for Fast 4D Reconstruction of Deformable Tissues

arXiv: https://arxiv.org/abs/2305.19906 · HTML: https://mlsanity.com/paper/2305.19906/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 31 May 2023.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [Baking Neural Radiance Fields for Real-Time View Synthesis](https://mlsanity.com/paper/2103.14645/) (2021)
- [Fast Dynamic Radiance Fields with Time-Aware Neural Voxels](https://mlsanity.com/paper/2205.15285/) (2022)
- [Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2011.02910/) (2020)
- [Neural Importance Sampling](https://mlsanity.com/paper/1808.03856/) (2018)
- [VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids](https://mlsanity.com/paper/2206.07695/) (2022)
- [Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery](https://mlsanity.com/paper/2206.15255/) (2022)
- [MedNeRF: Medical Neural Radiance Fields for Reconstructing 3D-aware CT-Projections from a Single X-ray](https://mlsanity.com/paper/2202.01020/) (2022)
- [E-DSSR: Efficient Dynamic Surgical Scene Reconstruction with Transformer-based Stereoscopic Depth Perception](https://mlsanity.com/paper/2107.00229/) (2021)

## Built on by (most cited first, 30 of 44)

- [A Survey on 3D Gaussian Splatting](https://mlsanity.com/paper/2401.03890/) (2024)
- [IGEV++: Iterative Multi-range Geometry Encoding Volumes for Stereo Matching](https://mlsanity.com/paper/2409.00638/) (2024)
- [Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting](https://mlsanity.com/paper/2401.16416/) (2024)
- [Deform3DGS: Flexible Deformation for Fast Surgical Scene Reconstruction with Gaussian Splatting](https://mlsanity.com/paper/2405.17835/) (2024)
- [Tracking and Mapping in Medical Computer Vision: A Review](https://mlsanity.com/paper/2310.11475/) (2023)
- [EndoGS: Deformable Endoscopic Tissues Reconstruction with Gaussian Splatting](https://mlsanity.com/paper/2401.11535/) (2024)
- [EndoGSLAM: Real-Time Dense Reconstruction and Tracking in Endoscopic Surgeries using Gaussian Splatting](https://mlsanity.com/paper/2403.15124/) (2024)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [EndoGaussian: Real-time Gaussian Splatting for Dynamic Endoscopic Scene Reconstruction](https://mlsanity.com/paper/2401.12561/) (2024)
- [LGS: A Light-weight 4D Gaussian Splatting for Efficient Surgical Scene Reconstruction](https://mlsanity.com/paper/2406.16073/) (2024)
- [Efficient Deformable Tissue Reconstruction via Orthogonal Neural Plane](https://mlsanity.com/paper/2312.15253/) (2023)
- [Learning 3D Gaussians for Extremely Sparse-View Cone-Beam CT Reconstruction](https://mlsanity.com/paper/2407.01090/) (2024)
- [EndoSparse: Real-Time Sparse View Synthesis of Endoscopic Scenes using Gaussian Splatting](https://mlsanity.com/paper/2407.01029/) (2024)
- [SurgicalGS: Dynamic 3D Gaussian Splatting for Accurate Robotic-Assisted Surgical Scene Reconstruction](https://mlsanity.com/paper/2410.09292/) (2024)
- [HFGS: 4D Gaussian Splatting with Emphasis on Spatial and Temporal High-Frequency Components for Endoscopic Scene Reconstruction](https://mlsanity.com/paper/2405.17872/) (2024)
- [SurgicalGaussian: Deformable 3D Gaussians for High-Fidelity Surgical Scene Reconstruction](https://mlsanity.com/paper/2407.05023/) (2024)
- [Advancing Dense Endoscopic Reconstruction with Gaussian Splatting-driven Surface Normal-aware Tracking and Mapping](https://mlsanity.com/paper/2501.19319/) (2025)
- [Online 3D reconstruction and dense tracking in endoscopic videos](https://mlsanity.com/paper/2409.06037/) (2024)
- [Surgical Gaussian Surfels: Highly Accurate Real-time Surgical Scene Rendering using Gaussian Surfels](https://mlsanity.com/paper/2503.04079/) (2025)
- [TOGS: Gaussian Splatting with Temporal Opacity Offset for Real-Time 4D DSA Rendering](https://mlsanity.com/paper/2403.19586/) (2024)
- [A Review of 3D Reconstruction Techniques for Deformable Tissues in Robotic Surgery](https://mlsanity.com/paper/2408.04426/) (2024)
- [Deformable Gaussian Splatting for Efficient and High-Fidelity Reconstruction of Surgical Scenes](https://mlsanity.com/paper/2501.01101/) (2025)
- [TiAVox: Time-aware Attenuation Voxels for Sparse-view 4D DSA Reconstruction](https://mlsanity.com/paper/2309.02318/) (2023)
- [Instrument-Splatting: Controllable Photorealistic Reconstruction of Surgical Instruments Using Gaussian Splatting](https://mlsanity.com/paper/2503.04082/) (2025)
- [EndoWave: 4D Gaussian Splatting with Rational Wavelet for Endoscopic Reconstruction](https://mlsanity.com/paper/2510.23087/) (2025)
- [EndoGaussians: Single View Dynamic Gaussian Splatting for Deformable Endoscopic Tissues Reconstruction](https://mlsanity.com/paper/2401.13352/) (2024)
- [FLex: Joint Pose and Dynamic Radiance Fields Optimization for Stereo Endoscopic Videos](https://mlsanity.com/paper/2403.12198/) (2024)
- [SimEndoGS: Efficient Data-driven Scene Simulation using Robotic Surgery Videos via Physics-embedded 3D Gaussians](https://mlsanity.com/paper/2405.00956/) (2024)
- [ColonNeRF: High-Fidelity Neural Reconstruction of Long Colonoscopy](https://mlsanity.com/paper/2312.02015/) (2023)
- [Neural Fields for 3D Tracking of Anatomy and Surgical Instruments in Monocular Laparoscopic Video Clips](https://mlsanity.com/paper/2403.19265/) (2024)

## Abstract (arXiv)

Reconstructing deformable tissues from endoscopic stereo videos in robotic surgery is crucial for various clinical applications. However, existing methods relying only on implicit representations are computationally expensive and require dozens of hours, which limits further practical applications. To address this challenge, we introduce LerPlane, a novel method for fast and accurate reconstruction of surgical scenes under a single-viewpoint setting. LerPlane treats surgical procedures as 4D volumes and factorizes them into explicit 2D planes of static and dynamic fields, leading to a compact memory footprint and significantly accelerated optimization. The efficient factorization is accomplished by fusing features obtained through linear interpolation of each plane and enables using lightweight neural networks to model surgical scenes. Besides, LerPlane shares static fields, significantly reducing the workload of dynamic tissue modeling. We also propose a novel sample scheme to boost optimization and improve performance in regions with tool occlusion and large motions. Experiments on DaVinci robotic surgery videos demonstrate that LerPlane accelerates optimization by over 100× while maintaining high quality across various non-rigid deformations, showing significant promise for future intraoperative surgery applications.