# Unifying Correspondence, Pose and NeRF for Pose-Free Novel View Synthesis from Stereo Pairs

arXiv: https://arxiv.org/abs/2312.07246 · HTML: https://mlsanity.com/paper/2312.07246/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 73 references that are in the map.
- First posted on arXiv 12 Dec 2023.

## Builds on (most cited first, 30 of 50)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2103.15595/) (2021)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021)
- [GMFlow: Learning Optical Flow via Global Matching](https://mlsanity.com/paper/2111.13680/) (2021)
- [LIFT: Learned Invariant Feature Transform](https://mlsanity.com/paper/1603.09114/) (2016)
- [FlowFormer: A Transformer Architecture for Optical Flow](https://mlsanity.com/paper/2203.16194/) (2022)
- [DSAC - Differentiable RANSAC for Camera Localization](https://mlsanity.com/paper/1611.05705/) (2016)
- [Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image](https://mlsanity.com/paper/2012.09855/) (2020)
- [NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior](https://mlsanity.com/paper/2212.07388/) (2022)
- [Deep Global Registration](https://mlsanity.com/paper/2004.11540/) (2020)
- [CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2303.11797/) (2023)
- [Self-Calibrating Neural Radiance Fields](https://mlsanity.com/paper/2108.13826/) (2021)
- [UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss](https://mlsanity.com/paper/1711.07837/) (2017)
- [GeoNeRF: Generalizing NeRF with Geometry Priors](https://mlsanity.com/paper/2111.13539/) (2021)
- [SPARF: Neural Radiance Fields from Sparse and Noisy Poses](https://mlsanity.com/paper/2211.11738/) (2022)
- [Learning to Find Good Correspondences](https://mlsanity.com/paper/1711.05971/) (2017)
- [MAGSAC++, a fast, reliable and accurate robust estimator](https://mlsanity.com/paper/1912.05909/) (2019)
- [MAGSAC: marginalizing sample consensus](https://mlsanity.com/paper/1803.07469/) (2018)
- [Generalizable Patch-Based Neural Rendering](https://mlsanity.com/paper/2207.10662/) (2022)
- [Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation](https://mlsanity.com/paper/2207.10866/) (2022)

## Built on by (most cited first, 30 of 37)

- [No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images](https://mlsanity.com/paper/2410.24207/) (2024)
- [MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Seconds](https://mlsanity.com/paper/2412.06974/) (2024)
- [FreeSplatter: Pose-free Gaussian Splatting for Sparse-view 3D Reconstruction](https://mlsanity.com/paper/2412.09573/) (2024)
- [Visual Representation Alignment for Multimodal Large Language Models](https://mlsanity.com/paper/2509.07979/) (2025)
- [No Pose at All: Self-Supervised Pose-Free 3D Gaussian Splatting from Sparse Views](https://mlsanity.com/paper/2508.01171/) (2025)
- [Emergent Temporal Correspondences from Video Diffusion Transformers](https://mlsanity.com/paper/2506.17220/) (2025)
- [Cross-View Completion Models are Zero-shot Correspondence Estimators](https://mlsanity.com/paper/2412.09072/) (2024)
- [PreF3R: Pose-Free Feed-Forward 3D Gaussian Splatting from Variable-length Image Sequence](https://mlsanity.com/paper/2411.16877/) (2024)
- [Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Survey](https://mlsanity.com/paper/2507.14501/) (2025)
- [Emergent Outlier View Rejection in Visual Geometry Grounded Transformers](https://mlsanity.com/paper/2512.04012/) (2025)
- [C3G: Learning Compact 3D Representations with 2K Gaussians](https://mlsanity.com/paper/2512.04021/) (2025)
- [SpatialCrafter: Unleashing the Imagination of Video Diffusion Models for Scene Reconstruction from Limited Observations](https://mlsanity.com/paper/2505.11992/) (2025)
- [VicaSplat: A Single Run is All You Need for 3D Gaussian Splatting and Camera Estimation from Unposed Video Frames](https://mlsanity.com/paper/2503.10286/) (2025)
- [TokenSplat: Token-aligned 3D Gaussian Splatting for Feed-forward Pose-free Reconstruction](https://mlsanity.com/paper/2603.00697/) (2026)
- [RegGS: Unposed Sparse Views Gaussian Splatting with 3DGS Registration](https://mlsanity.com/paper/2507.08136/) (2025)
- [3D Scene Prompting for Scene-Consistent Camera-Controllable Video Generation](https://mlsanity.com/paper/2510.14945/) (2025)
- [D2USt3R: Enhancing 3D Reconstruction for Dynamic Scenes](https://mlsanity.com/paper/2504.06264/) (2025)
- [EcoSplat: Efficiency-controllable Feed-forward 3D Gaussian Splatting from Multi-view Images](https://mlsanity.com/paper/2512.18692/) (2025)
- [LagerNVS: Latent Geometry for Fully Neural Real-time Novel View Synthesis](https://mlsanity.com/paper/2603.20176/) (2026)
- [Vid-CamEdit: Video Camera Trajectory Editing with Generative Rendering from Estimated Geometry](https://mlsanity.com/paper/2506.13697/) (2025)
- [SPFSplatV2: Efficient Self-Supervised Pose-Free 3D Gaussian Splatting from Sparse Views](https://mlsanity.com/paper/2509.17246/) (2025)
- [Text-Aware Image Restoration with Diffusion Models](https://mlsanity.com/paper/2506.09993/) (2025)
- [S^4M: Boosting Semi-Supervised Instance Segmentation with SAM](https://mlsanity.com/paper/2504.05301/) (2025)
- [SaLon3R: Structure-aware Long-term Generalizable 3D Reconstruction from Unposed Images](https://mlsanity.com/paper/2510.15072/) (2025)
- [TORA: Topological Representation Alignment for 3D Shape Assembly](https://mlsanity.com/paper/2604.04050/) (2026)
- [Recollection from Pensieve: Novel View Synthesis via Learning from Uncalibrated Videos](https://mlsanity.com/paper/2505.13440/) (2025)
- [Stylos: Multi-View 3D Stylization with Single-Forward Gaussian Splatting](https://mlsanity.com/paper/2509.26455/) (2025)
- [G3Splat: Geometrically Consistent Generalizable Gaussian Splatting](https://mlsanity.com/paper/2512.17547/) (2025)
- [ViewSplat: View-Adaptive 3D Gaussian Splatting for Feed-Forward Synthesis](https://mlsanity.com/paper/2603.25265/) (2026)
- [From None to All: Self-Supervised 3D Reconstruction via Novel View Synthesis](https://mlsanity.com/paper/2603.27455/) (2026)

## Abstract (arXiv)

This work delves into the task of pose-free novel view synthesis from stereo pairs, a challenging and pioneering task in 3D vision. Our innovative framework, unlike any before, seamlessly integrates 2D correspondence matching, camera pose estimation, and NeRF rendering, fostering a synergistic enhancement of these tasks. We achieve this through designing an architecture that utilizes a shared representation, which serves as a foundation for enhanced 3D geometry understanding. Capitalizing on the inherent interplay between the tasks, our unified framework is trained end-to-end with the proposed training strategy to improve overall model accuracy. Through extensive evaluations across diverse indoor and outdoor scenes from two real-world datasets, we demonstrate that our approach achieves substantial improvement over previous methodologies, especially in scenarios characterized by extreme viewpoint changes and the absence of accurate camera poses.