# G3R: Gradient Guided Generalizable Reconstruction

arXiv: https://arxiv.org/abs/2409.19405 · HTML: https://mlsanity.com/paper/2409.19405/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 85 of its 94 references that are in the map.
- First posted on arXiv 28 Sep 2024.

## Builds on (most cited first, 30 of 85)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023)
- [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2103.15595/) (2021)
- [Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis](https://mlsanity.com/paper/2308.09713/) (2023)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction](https://mlsanity.com/paper/2312.12337/) (2023)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering](https://mlsanity.com/paper/2311.12775/) (2023)
- [BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks](https://mlsanity.com/paper/1911.10127/) (2019)
- [Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs](https://mlsanity.com/paper/2112.10703/) (2021)
- [Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model](https://mlsanity.com/paper/2311.06214/) (2023)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes](https://mlsanity.com/paper/2312.07920/) (2023)
- [GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting](https://mlsanity.com/paper/2404.19702/) (2024)
- [Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering](https://mlsanity.com/paper/2106.02634/) (2021)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [Neural Scene Graphs for Dynamic Scenes](https://mlsanity.com/paper/2011.10379/) (2020)

## Built on by (most cited first, 30 of 31)

- [SCube: Instant Large-Scale Scene Reconstruction using VoxSplats](https://mlsanity.com/paper/2410.20030/) (2024)
- [Feed-Forward Bullet-Time Reconstruction of Dynamic Scenes from Monocular Videos](https://mlsanity.com/paper/2412.03526/) (2024)
- [Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Survey](https://mlsanity.com/paper/2507.14501/) (2025)
- [EVolSplat: Efficient Volume-based Gaussian Splatting for Urban View Synthesis](https://mlsanity.com/paper/2503.20168/) (2025)
- [iLRM: An Iterative Large 3D Reconstruction Model](https://mlsanity.com/paper/2507.23277/) (2025)
- [ReSplat: Learning Recurrent Gaussian Splatting](https://mlsanity.com/paper/2510.08575/) (2025)
- [TokenGS: Decoupling 3D Gaussian Prediction from Pixels with Learnable Tokens](https://mlsanity.com/paper/2604.15239/) (2026)
- [4Real-Video-V2: Fused View-Time Attention and Feedforward Reconstruction for 4D Scene Generation](https://mlsanity.com/paper/2506.18839/) (2025)
- [Test-Time Training with KV Binding Is Secretly Linear Attention](https://mlsanity.com/paper/2602.21204/) (2026)
- [QuickSplat: Fast 3D Surface Reconstruction via Learned Gaussian Initialization](https://mlsanity.com/paper/2505.05591/) (2025)
- [PINGS: Gaussian Splatting Meets Distance Fields within a Point-Based Implicit Neural Map](https://mlsanity.com/paper/2502.05752/) (2025)
- [Flux4D: Flow-based Unsupervised 4D Reconstruction](https://mlsanity.com/paper/2512.03210/) (2025)
- [Distractor-free Generalizable 3D Gaussian Splatting](https://mlsanity.com/paper/2411.17605/) (2024)
- [AnchorSplat: Feed-Forward 3D Gaussian Splatting with 3D Geometric Priors](https://mlsanity.com/paper/2604.07053/) (2026)
- [IDESplat: Iterative Depth Probability Estimation for Generalizable 3D Gaussian Splatting](https://mlsanity.com/paper/2601.03824/) (2026)
- [GenAssets: Generating in-the-wild 3D Assets in Latent Space](https://mlsanity.com/paper/2604.23010/) (2026)
- [SaLF: Sparse Local Fields for Multi-Sensor Rendering in Real-Time](https://mlsanity.com/paper/2507.18713/) (2025)
- [GaussianLens: Localized High-Resolution Reconstruction via On-Demand Gaussian Densification](https://mlsanity.com/paper/2509.25603/) (2025)
- [IDSplat: Instance-Decomposed 3D Gaussian Splatting for Driving Scenes](https://mlsanity.com/paper/2511.19235/) (2025)
- [UFO: Unifying Feed-Forward and Optimization-based Methods for Large Driving Scene Modeling](https://mlsanity.com/paper/2602.20943/) (2026)
- [BADGR: Bundle Adjustment Diffusion Conditioned by GRadients for Wide-Baseline Floor Plan Reconstruction](https://mlsanity.com/paper/2503.19340/) (2025)
- [FreeFix: Boosting 3D Gaussian Splatting via Fine-Tuning-Free Diffusion Models](https://mlsanity.com/paper/2601.20857/) (2026)
- [SurfSplat: Conquering Feedforward 2D Gaussian Splatting with Surface Continuity Priors](https://mlsanity.com/paper/2602.02000/) (2026)
- [Diff3R: Feed-forward 3D Gaussian Splatting with Uncertainty-aware Differentiable Optimization](https://mlsanity.com/paper/2604.01030/) (2026)
- [Learn2Splat: Extending the Horizon of Learned 3DGS Optimization](https://mlsanity.com/paper/2605.15760/) (2026)
- [Diffusion-guided Generalizable Enhancer for Urban Scene Reconstruction](https://mlsanity.com/paper/2605.22420/) (2026)
- [ADGaussian: Generalizable Gaussian Splatting for Autonomous Driving via Multi-modal Joint Learning](https://mlsanity.com/paper/2504.00437/) (2025)
- [FlowBender: Feedback-Aware Training for Self-Correcting Conditional Flows](https://mlsanity.com/paper/2606.20404/) (2026)
- [L2D2-GS: Learning to Densify for Feedforward Dynamic Gaussian Scene Reconstruction](https://mlsanity.com/paper/2606.29374/) (2026)
- [Into the LidarVerse: A Real-Time Unified LiDAR World Model Across Tasks and Sensors](https://mlsanity.com/paper/2610.19201/) (2026)

## Abstract (arXiv)

Large scale 3D scene reconstruction is important for applications such as virtual reality and simulation. Existing neural rendering approaches (e.g., NeRF, 3DGS) have achieved realistic reconstructions on large scenes, but optimize per scene, which is expensive and slow, and exhibit noticeable artifacts under large view changes due to overfitting. Generalizable approaches or large reconstruction models are fast, but primarily work for small scenes/objects and often produce lower quality rendering results. In this work, we introduce G3R, a generalizable reconstruction approach that can efficiently predict high-quality 3D scene representations for large scenes. We propose to learn a reconstruction network that takes the gradient feedback signals from differentiable rendering to iteratively update a 3D scene representation, combining the benefits of high photorealism from per-scene optimization with data-driven priors from fast feed-forward prediction methods. Experiments on urban-driving and drone datasets show that G3R generalizes across diverse large scenes and accelerates the reconstruction process by at least 10x while achieving comparable or better realism compared to 3DGS, and also being more robust to large view changes.