# A Construct-Optimize Approach to Sparse View Synthesis without Camera Pose

arXiv: https://arxiv.org/abs/2405.03659 · HTML: https://mlsanity.com/paper/2405.03659/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 51 references that are in the map.
- First posted on arXiv 6 May 2024.

## Builds on (most cited first, 22 of 22)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019)
- [SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering](https://mlsanity.com/paper/2311.12775/) (2023)
- [RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs](https://mlsanity.com/paper/2112.00724/) (2021)
- [FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting](https://mlsanity.com/paper/2312.00451/) (2023)
- [NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior](https://mlsanity.com/paper/2212.07388/) (2022)
- [Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP](https://mlsanity.com/paper/2308.02487/) (2023)
- [MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimation](https://mlsanity.com/paper/2307.14460/) (2023)
- [Depth-Regularized Optimization for 3D Gaussian Splatting in Few-Shot Images](https://mlsanity.com/paper/2311.13398/) (2023)
- [GNeRF: GAN-based Neural Radiance Field without Posed Camera](https://mlsanity.com/paper/2103.15606/) (2021)
- [Learning-Based View Synthesis for Light Field Cameras](https://mlsanity.com/paper/1609.02974/) (2016)
- [Is Attention All That NeRF Needs?](https://mlsanity.com/paper/2207.13298/) (2022)
- [Progressively Optimized Local Radiance Fields for Robust View Synthesis](https://mlsanity.com/paper/2303.13791/) (2023)
- [CamP: Camera Preconditioning for Neural Radiance Fields](https://mlsanity.com/paper/2308.10902/) (2023)
- [Robust Camera Pose Refinement for Multi-Resolution Hash Encoding](https://mlsanity.com/paper/2302.01571/) (2023)

## Built on by (most cited first, 25 of 25)

- [MAtCha Gaussians: Atlas of Charts for High-Quality Geometry and Photorealism From Sparse Views](https://mlsanity.com/paper/2412.06767/) (2024)
- [SfM-Free 3D Gaussian Splatting via Hierarchical Training](https://mlsanity.com/paper/2412.01553/) (2024)
- [Volumetrically Consistent 3D Gaussian Rasterization](https://mlsanity.com/paper/2412.03378/) (2024)
- [Off The Grid: Detection of Primitives for Feed-Forward 3D Gaussian Splatting](https://mlsanity.com/paper/2512.15508/) (2025)
- [RePoseD: Efficient Relative Pose Estimation With Known Depth Information](https://mlsanity.com/paper/2501.07742/) (2025)
- [Generic Objects as Pose Probes for Few-shot View Synthesis](https://mlsanity.com/paper/2408.16690/) (2024)
- [Correspondence-Guided SfM-Free 3D Gaussian Splatting for NVS](https://mlsanity.com/paper/2408.08723/) (2024)
- [Gaussian Scenes: Pose-Free Sparse-View Scene Reconstruction using Depth-Enhanced Diffusion Priors](https://mlsanity.com/paper/2411.15966/) (2024)
- [VideoLifter: Lifting Videos to 3D with Fast Hierarchical Stereo Alignment](https://mlsanity.com/paper/2501.01949/) (2025)
- [Unposed 3DGS Reconstruction with Probabilistic Procrustes Mapping](https://mlsanity.com/paper/2507.18541/) (2025)
- [AD-GS: Alternating Densification for Sparse-Input 3D Gaussian Splatting](https://mlsanity.com/paper/2509.11003/) (2025)
- [KFC-W: Generating 3D-Consistent Videos from Unposed Internet Photos](https://mlsanity.com/paper/2411.13549/) (2024)
- [ActiveInitSplat: How Active Image Selection Helps Gaussian Splatting](https://mlsanity.com/paper/2503.06859/) (2025)
- [Generating 360deg Video is What You Need For a 3D Scene](https://mlsanity.com/paper/2504.02045/) (2025)
- [MS-GS: Multi-Appearance Sparse-View 3D Gaussian Splatting in the Wild](https://mlsanity.com/paper/2509.15548/) (2025)
- [TrackGS: Optimizing COLMAP-Free 3D Gaussian Splatting with Global Track Constraints](https://mlsanity.com/paper/2502.19800/) (2025)
- [Recollection from Pensieve: Novel View Synthesis via Learning from Uncalibrated Videos](https://mlsanity.com/paper/2505.13440/) (2025)
- [Repurposing Marigold for Zero-Shot Metric Depth Estimation via Defocus Blur Cues](https://mlsanity.com/paper/2505.17358/) (2025)
- [Immediate 3D Gaussian Splat Reconstruction of Unordered Input with Global Consistency](https://mlsanity.com/paper/2607.14481/) (2026)
- [Sampling for View Synthesis: From Local Light Field Fusion to Neural Radiance Fields and Beyond](https://mlsanity.com/paper/2408.04586/) (2024)
- [PMNI: Pose-free Multi-view Normal Integration for Reflective and Textureless Surface Reconstruction](https://mlsanity.com/paper/2504.08410/) (2025)
- [LighthouseGS: Indoor Structure-aware 3D Gaussian Splatting for Panorama-Style Mobile Captures](https://mlsanity.com/paper/2507.06109/) (2025)
- [Pseudo-View Enhancement via Confidence Fusion for Unposed Sparse-View Reconstruction](https://mlsanity.com/paper/2602.21535/) (2026)
- [DMAligner: Enhancing Image Alignment via Diffusion Model Based View Synthesis](https://mlsanity.com/paper/2602.23022/) (2026)
- [Improving Novel view synthesis of 360^circ Scenes in Extremely Sparse Views by Jointly Training Hemisphere Sampled Synthetic Images](https://mlsanity.com/paper/2505.19264/) (2025)

## Abstract (arXiv)

Novel view synthesis from a sparse set of input images is a challenging problem of great practical interest, especially when camera poses are absent or inaccurate. Direct optimization of camera poses and usage of estimated depths in neural radiance field algorithms usually do not produce good results because of the coupling between poses and depths, and inaccuracies in monocular depth estimation. In this paper, we leverage the recent 3D Gaussian splatting method to develop a novel construct-and-optimize method for sparse view synthesis without camera poses. Specifically, we construct a solution progressively by using monocular depth and projecting pixels back into the 3D world. During construction, we optimize the solution by detecting 2D correspondences between training views and the corresponding rendered images. We develop a unified differentiable pipeline for camera registration and adjustment of both camera poses and depths, followed by back-projection. We also introduce a novel notion of an expected surface in Gaussian splatting, which is critical to our optimization. These steps enable a coarse solution, which can then be low-pass filtered and refined using standard optimization methods. We demonstrate results on the Tanks and Temples and Static Hikes datasets with as few as three widely-spaced views, showing significantly better quality than competing methods, including those with approximate camera pose information. Moreover, our results improve with more views and outperform previous InstantNGP and Gaussian Splatting algorithms even when using half the dataset. Project page: https://raymondjiangkw.github.io/cogs.github.io/