# Correspondence-Guided SfM-Free 3D Gaussian Splatting for NVS

arXiv: https://arxiv.org/abs/2408.08723 · HTML: https://mlsanity.com/paper/2408.08723/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 39 references that are in the map.
- First posted on arXiv 16 Aug 2024.

## Builds on (most cited first, 28 of 28)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis](https://mlsanity.com/paper/2308.09713/) (2023)
- [Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields](https://mlsanity.com/paper/2304.06706/) (2023)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM](https://mlsanity.com/paper/2312.02126/) (2023)
- [GS-SLAM: Dense Visual SLAM with 3D Gaussian Splatting](https://mlsanity.com/paper/2311.11700/) (2023)
- [Single-View View Synthesis with Multiplane Images](https://mlsanity.com/paper/2004.11364/) (2020)
- [Free View Synthesis](https://mlsanity.com/paper/2008.05511/) (2020)
- [Depth-Regularized Optimization for 3D Gaussian Splatting in Few-Shot Images](https://mlsanity.com/paper/2311.13398/) (2023)
- [Stable View Synthesis](https://mlsanity.com/paper/2011.07233/) (2020)
- [MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesis](https://mlsanity.com/paper/2103.14910/) (2021)
- [QuadTree Attention for Vision Transformers](https://mlsanity.com/paper/2201.02767/) (2022)
- [Progressively Optimized Local Radiance Fields for Robust View Synthesis](https://mlsanity.com/paper/2303.13791/) (2023)
- [Differentiable Point-Based Radiance Fields for Efficient View Synthesis](https://mlsanity.com/paper/2205.14330/) (2022)
- [Neural Point Catacaustics for Novel-View Synthesis of Reflections](https://mlsanity.com/paper/2301.01087/) (2023)
- [Worldsheet: Wrapping the World in a 3D Sheet for View Synthesis from a Single Image](https://mlsanity.com/paper/2012.09854/) (2020)
- [MonoNeRF: Learning a Generalizable Dynamic Radiance Field from Monocular Videos](https://mlsanity.com/paper/2212.13056/) (2022)
- [RUST: Latent Neural Scene Representations from Unposed Imagery](https://mlsanity.com/paper/2211.14306/) (2022)
- [FlowCam: Training Generalizable 3D Radiance Fields without Camera Poses via Pixel-Aligned Scene Flow](https://mlsanity.com/paper/2306.00180/) (2023)
- [Video Autoencoder: self-supervised disentanglement of static 3D structure and motion](https://mlsanity.com/paper/2110.02951/) (2021)
- [A Construct-Optimize Approach to Sparse View Synthesis without Camera Pose](https://mlsanity.com/paper/2405.03659/) (2024)

## Built on by (most cited first, 8 of 8)

- [3R-GS: Best Practice in Optimizing Camera Poses Along with 3DGS](https://mlsanity.com/paper/2504.04294/) (2025)
- [CAGS: Open-Vocabulary 3D Scene Understanding with Context-Aware Gaussian Splatting](https://mlsanity.com/paper/2504.11893/) (2025)
- [Dust to Tower: Coarse-to-Fine Photo-Realistic Scene Reconstruction from Sparse Uncalibrated Images](https://mlsanity.com/paper/2412.19518/) (2024)
- [ActiveInitSplat: How Active Image Selection Helps Gaussian Splatting](https://mlsanity.com/paper/2503.06859/) (2025)
- [GeoAvatar: Adaptive Geometrical Gaussian Splatting for 3D Head Avatar](https://mlsanity.com/paper/2507.18155/) (2025)
- [UVRM: A Scalable 3D Reconstruction Model from Unposed Videos](https://mlsanity.com/paper/2501.09347/) (2025)
- [Pose-Free Omnidirectional Gaussian Splatting for 360-Degree Videos with Consistent Depth Priors](https://mlsanity.com/paper/2603.23324/) (2026)
- [Reliability-Regulated Trajectory Optimization for Progressive COLMAP-Free 3D Gaussian Splatting](https://mlsanity.com/paper/2609.30865/) (2026)

## Abstract (arXiv)

Novel View Synthesis (NVS) without Structure-from-Motion (SfM) pre-processed camera poses--referred to as SfM-free methods--is crucial for promoting rapid response capabilities and enhancing robustness against variable operating conditions. Recent SfM-free methods have integrated pose optimization, designing end-to-end frameworks for joint camera pose estimation and NVS. However, most existing works rely on per-pixel image loss functions, such as L2 loss. In SfM-free methods, inaccurate initial poses lead to misalignment issue, which, under the constraints of per-pixel image loss functions, results in excessive gradients, causing unstable optimization and poor convergence for NVS. In this study, we propose a correspondence-guided SfM-free 3D Gaussian splatting for NVS. We use correspondences between the target and the rendered result to achieve better pixel alignment, facilitating the optimization of relative poses between frames. We then apply the learned poses to optimize the entire scene. Each 2D screen-space pixel is associated with its corresponding 3D Gaussians through approximated surface rendering to facilitate gradient back propagation. Experimental results underline the superior performance and time efficiency of the proposed approach compared to the state-of-the-art baselines.