Two-View Accumulation as the Primary Training Lever for Hybrid-Capture Gaussian Splatting: A Variance-Decomposition View of When Gradient Surgery Helps
Not on the map. Why?
Abstract (arXiv)
Hybrid-capture novel view synthesis combines images at substantially different camera distances (e.g., aerial drone and ground-level views). Standard 3D Gaussian Splatting (3DGS), trained for 30K iterations with one rendered view per optimizer step, under-fits the minority regime by 1-3 dB on five hybrid-capture benchmarks. We isolate the lever that closes this gap. Among compute-matched alternatives -- vanilla 60K iterations, magnitude corrections (GradNorm), direction-aware near/far gradient surgery, projective preconditioning, confidence-gated sample-level surgery, and a random two-view-per-step control -- the simplest structural change wins: rendering two views per optimizer step. The pairing rule (geometry-defined near/far, random, or active loss-disparity) does not change PSNR beyond seed variance on any of the five scenes; the structural change of having two views per step does. We propose a variance-decomposition framework that predicts and explains this finding: under bimodal camera regimes, between-regime gradient variance turns out to be small relative to within-regime variance in 3DGS, so structured and random pairings are variance-equivalent in expectation, and the variance halving from two-view accumulation itself is the dominant effect. We verify the framework on five scenes whose camera-altitude bimodality coefficients span [0.55, 1.00], and we report the negative result that direction-aware projection, magnitude correction, confidence gating, and an active loss-disparity pairing all fall within seed variance of random two-view pairing. The two-view structural lever transfers cleanly to the Scaffold-GS and Pixel-GS backbones. We position this work as an honest characterization of which training-side axes do and do not move PSNR for hybrid-capture 3DGS, together with the framework that explains why.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 29 references that are in the map. First posted on arXiv 29 Apr 2026.
Builds on
27 of 29 references.
References · 27
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
- 2020Gradient Surgery for Multi-Task Learningcited by 1,256
- 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
- 2023Scaffold-GS: Structured 3D Gaussians for View-Adaptive Renderingcited by 695
- 2023Mip-Splatting: Alias-free 3D Gaussian Splattingcited by 667
- 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
- 2024Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussianscited by 230
- 2024A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasetscited by 216
- 2024VastGaussian: Vast 3D Gaussians for Large Scene Reconstructioncited by 209
- 2024CityGaussian: Real-time High-quality Large-Scale Scene Rendering with Gaussianscited by 192
- 2023MatrixCity: A Large-scale City Dataset for City-scale Neural Rendering and Beyondcited by 185
- 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 169
15 more
- 2024WildGaussians: 3D Gaussian Splatting in the Wildcited by 152
- 2023Multi-Scale 3D Gaussian Splatting for Anti-Aliased Renderingcited by 138
- 2024Revising Densification in Gaussian Splattingcited by 128
- 2024Pixel-GS: Density Control with Pixel-aware Gradient for 3D Gaussian Splattingcited by 113
- 2024AbsGS: Recovering Fine Details for 3D Gaussian Splattingcited by 82
- 2024Analytic-Splatting: Anti-Aliased 3D Gaussian Splatting via Analytic Integrationcited by 67
- 2024GeoGaussian: Geometry-aware Gaussian Splatting for Scene Renderingcited by 52
- 2024Horizon-GS: Unified 3D Gaussian Splatting for Large-Scale Aerial-to-Ground Scenescited by 30
- 2023Floaters No More: Radiance Field Gradient Scaling for Improved Near-Camera Trainingcited by 25
- 2024HO-Gaussian: Hybrid Optimization of 3D Gaussian Splatting for Urban Scenescited by 16
- 2024HybridGS: Decoupling Transients and Statics with 2D and 3D Gaussian Splattingcited by 16
- 2024HGS-Mapping: Online Dense Mapping Using Hybrid Gaussian Representation in Urban Scenescited by 11
- 2025CrossView-GS: Cross-view Gaussian Splatting For Large-scale Scene Reconstructioncited by 5
- 2025DC4GS: Directional Consistency-Driven Adaptive Density Control for 3D Gaussian Splattingcited by 2
- 2025EGGS: Exchangeable 2D/3D Gaussian Splatting for Geometry-Appearance Balanced Novel View Synthesiscited by 2
Built on by
Not cited yet.