# On-the-fly Reconstruction for Large-Scale Novel View Synthesis from Unposed Images

arXiv: https://arxiv.org/abs/2506.05558 · HTML: https://mlsanity.com/paper/2506.05558/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 68 references that are in the map.
- First posted on arXiv 5 Jun 2025.

## Builds on (most cited first, 30 of 32)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs](https://mlsanity.com/paper/2112.10703/) (2021)
- [SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM](https://mlsanity.com/paper/2312.02126/) (2023)
- [Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians](https://mlsanity.com/paper/2403.17898/) (2024)
- [A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets](https://mlsanity.com/paper/2406.12080/) (2024)
- [Mini-Splatting: Representing Scenes with a Constrained Number of Gaussians](https://mlsanity.com/paper/2403.14166/) (2024)
- [Self-Calibrating Neural Radiance Fields](https://mlsanity.com/paper/2108.13826/) (2021)
- [Taming 3DGS: High-Quality Radiance Fields with Limited Resources](https://mlsanity.com/paper/2406.15643/) (2024)
- [CityGaussian: Real-time High-quality Large-Scale Scene Rendering with Gaussians](https://mlsanity.com/paper/2404.01133/) (2024)
- [Robust Consistent Video Depth Estimation](https://mlsanity.com/paper/2012.05901/) (2020)
- [DeepV2D: Video to Depth with Differentiable Structure from Motion](https://mlsanity.com/paper/1812.04605/) (2018)
- [DeepFactors: Real-Time Probabilistic Dense Monocular SLAM](https://mlsanity.com/paper/2001.05049/) (2020)
- [XFeat: Accelerated Features for Lightweight Image Matching](https://mlsanity.com/paper/2404.19174/) (2024)
- [Progressively Optimized Local Radiance Fields for Robust View Synthesis](https://mlsanity.com/paper/2303.13791/) (2023)
- [Grid-guided Neural Radiance Fields for Large Urban Scenes](https://mlsanity.com/paper/2303.14001/) (2023)
- [ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wild](https://mlsanity.com/paper/2207.09137/) (2022)
- [Scene Coordinate Reconstruction: Posing of Image Collections via Incremental Learning of a Relocalizer](https://mlsanity.com/paper/2404.14351/) (2024)
- [How NeRFs and 3D Gaussian Splatting are Reshaping SLAM: a Survey](https://mlsanity.com/paper/2402.13255/) (2024)
- [SMERF: Streamable Memory Efficient Radiance Fields for Real-Time Large-Scene Exploration](https://mlsanity.com/paper/2312.07541/) (2023)
- [Multiview Stereo with Cascaded Epipolar RAFT](https://mlsanity.com/paper/2205.04502/) (2022)
- [High-Fidelity SLAM Using Gaussian Splatting with Rendering-Guided Densification and Regularized Optimization](https://mlsanity.com/paper/2403.12535/) (2024)

## Built on by (most cited first, 30 of 36)

- [AMB3R: Accurate Feed-forward Metric-scale 3D Reconstruction with Backend](https://mlsanity.com/paper/2511.20343/) (2025)
- [RealX3D: A Physically-Degraded 3D Benchmark for Multi-view Visual Restoration and Reconstruction](https://mlsanity.com/paper/2512.23437/) (2025)
- [Off The Grid: Detection of Primitives for Feed-Forward 3D Gaussian Splatting](https://mlsanity.com/paper/2512.15508/) (2025)
- [ARTDECO: Towards Efficient and High-Fidelity On-the-Fly 3D Reconstruction with Structured Scene Representation](https://mlsanity.com/paper/2510.08551/) (2025)
- [Changes in Real Time: Online Scene Change Detection with Multi-View Fusion](https://mlsanity.com/paper/2511.12370/) (2025)
- [On-the-fly Large-scale 3D Reconstruction from Multi-Camera Rigs](https://mlsanity.com/paper/2512.08498/) (2025)
- [PLANING: A Loosely Coupled Triangle-Gaussian Framework for Streaming 3D Reconstruction](https://mlsanity.com/paper/2601.22046/) (2026)
- [DiskChunGS: Large-Scale 3D Gaussian SLAM Through Chunk-Based Memory Management](https://mlsanity.com/paper/2511.23030/) (2025)
- [M^3: Dense Matching Meets Multi-View Foundation Models for Monocular Gaussian Splatting SLAM](https://mlsanity.com/paper/2603.16844/) (2026)
- [Face Anything: 4D Face Reconstruction from Any Image Sequence](https://mlsanity.com/paper/2604.19702/) (2026)
- [Immediate 3D Gaussian Splat Reconstruction of Unordered Input with Global Consistency](https://mlsanity.com/paper/2607.14481/) (2026)
- [Virtual Memory for 3D Gaussian Splatting](https://mlsanity.com/paper/2506.19415/) (2025)
- [LighthouseGS: Indoor Structure-aware 3D Gaussian Splatting for Panorama-Style Mobile Captures](https://mlsanity.com/paper/2507.06109/) (2025)
- [CoRe-GS: Coarse-to-Refined Gaussian Splatting with Semantic Object Focus](https://mlsanity.com/paper/2509.04859/) (2025)
- [A-TDOM: Active TDOM via On-the-Fly 3DGS](https://mlsanity.com/paper/2509.12759/) (2025)
- [PrismGS: Physically-Grounded Anti-Aliasing for High-Fidelity Large-Scale 3D Gaussian Splatting](https://mlsanity.com/paper/2510.07830/) (2025)
- [The Impact and Outlook of 3D Gaussian Splatting](https://mlsanity.com/paper/2510.26694/) (2025)
- [Hybrid Foveated Path Tracing with Peripheral Gaussians for Immersive Anatomy](https://mlsanity.com/paper/2601.22026/) (2026)
- [VistaGEN: Consistent Driving Video Generation with Fine-Grained Control Using Multiview Visual-Language Reasoning](https://mlsanity.com/paper/2603.28353/) (2026)
- [OF3GS: On-the-Fly Feed-Forward 3D Gaussian Splatting from Unposed Images](https://mlsanity.com/paper/2606.03254/) (2026)
- [Compression of 3D Gaussian Splatting Data Using GPU-friendly Graphics Texture Coding](https://mlsanity.com/paper/2607.14513/) (2026)
- [Splatblox: Traversability-Aware Gaussian Splatting for Outdoor Robot Navigation](https://mlsanity.com/paper/2511.18525/) (2025)
- [MOSAIC-GS: Monocular Scene Reconstruction via Advanced Initialization for Complex Dynamic Environments](https://mlsanity.com/paper/2601.05368/) (2026)
- [Splats in Splats++: Robust and Generalizable 3D Gaussian Splatting Steganography](https://mlsanity.com/paper/2604.15862/) (2026)
- [Faster 3D Gaussian Splatting Convergence via Structure-Aware Densification](https://mlsanity.com/paper/2604.28016/) (2026)
- [Remember to be Curious: Episodic Context and Persistent Worlds for 3D Exploration](https://mlsanity.com/paper/2605.22814/) (2026)
- [MoonSplat: Monocular Online Gaussian Splatting with Sim(3) Global Optimization](https://mlsanity.com/paper/2606.17935/) (2026)
- [Robust and Efficient Monocular 3D Gaussian SLAM for Kilometer-Scale Outdoor Scenes](https://mlsanity.com/paper/2606.30436/) (2026)
- [Online Neural Space Time Memory for Dynamic Novel View Synthesis](https://mlsanity.com/paper/2607.15271/) (2026)
- [Rendering 3D Gaussians on a Graph Processor](https://mlsanity.com/paper/2607.15951/) (2026)

## Abstract (arXiv)

Radiance field methods such as 3D Gaussian Splatting (3DGS) allow easy reconstruction from photos, enabling free-viewpoint navigation. Nonetheless, pose estimation using Structure from Motion and 3DGS optimization can still each take between minutes and hours of computation after capture is complete. SLAM methods combined with 3DGS are fast but struggle with wide camera baselines and large scenes. We present an on-the-fly method to produce camera poses and a trained 3DGS immediately after capture. Our method can handle dense and wide-baseline captures of ordered photo sequences and large-scale scenes. To do this, we first introduce fast initial pose estimation, exploiting learned features and a GPU-friendly mini bundle adjustment. We then introduce direct sampling of Gaussian primitive positions and shapes, incrementally spawning primitives where required, significantly accelerating training. These two efficient steps allow fast and robust joint optimization of poses and Gaussian primitives. Our incremental approach handles large-scale scenes by introducing scalable radiance field construction, progressively clustering 3DGS primitives, storing them in anchors, and offloading them from the GPU. Clustered primitives are progressively merged, keeping the required scale of 3DGS at any viewpoint. We evaluate our solution on a variety of datasets and show that our solution can provide on-the-fly processing of all the capture scenarios and scene sizes we target while remaining competitive with other methods that only handle specific capture styles or scene sizes in speed, image quality, or both.