# LU-NeRF: Scene and Pose Estimation by Synchronizing Local Unposed NeRFs

arXiv: https://arxiv.org/abs/2306.05410 · HTML: https://mlsanity.com/paper/2306.05410/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 69 references that are in the map.
- First posted on arXiv 8 Jun 2023.

## Builds on (most cited first, 30 of 44)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2103.15595/) (2021)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs](https://mlsanity.com/paper/2112.00724/) (2021)
- [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021)
- [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [iMAP: Implicit Mapping and Positioning in Real-Time](https://mlsanity.com/paper/2103.12352/) (2021)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [Deep ViT Features as Dense Visual Descriptors](https://mlsanity.com/paper/2112.05814/) (2021)
- [NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior](https://mlsanity.com/paper/2212.07388/) (2022)
- [Pixel-Perfect Structure-from-Motion with Featuremetric Refinement](https://mlsanity.com/paper/2108.08291/) (2021)
- [NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fields](https://mlsanity.com/paper/2210.13641/) (2022)
- [Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild](https://mlsanity.com/paper/1911.11130/) (2019)
- [A Survey of Structure from Motion](https://mlsanity.com/paper/1701.08493/) (2017)
- [Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotations](https://mlsanity.com/paper/2012.09988/) (2020)
- [GNeRF: GAN-based Neural Radiance Field without Posed Camera](https://mlsanity.com/paper/2103.15606/) (2021)
- [Progressively Optimized Local Radiance Fields for Robust View Synthesis](https://mlsanity.com/paper/2303.13791/) (2023)
- [Unsupervised Learning of Shape and Pose with Differentiable Point Clouds](https://mlsanity.com/paper/1810.09381/) (2018)

## Built on by (most cited first, 30 of 33)

- [COLMAP-Free 3D Gaussian Splatting](https://mlsanity.com/paper/2312.07504/) (2023)
- [Scene Coordinate Reconstruction: Posing of Image Collections via Incremental Learning of a Relocalizer](https://mlsanity.com/paper/2404.14351/) (2024)
- [InstantSplat: Sparse-view Gaussian Splatting in Seconds](https://mlsanity.com/paper/2403.20309/) (2024)
- [FlowMap: High-Quality Camera Poses, Intrinsics, and Depth via Gradient Descent](https://mlsanity.com/paper/2404.15259/) (2024)
- [LongSplat: Robust Unposed 3D Gaussian Splatting for Casual Long Videos](https://mlsanity.com/paper/2508.14041/) (2025)
- [StreamGS: Online Generalizable Gaussian Splatting Reconstruction for Unposed Image Streams](https://mlsanity.com/paper/2503.06235/) (2025)
- [ThermalNeRF: Thermal Radiance Fields](https://mlsanity.com/paper/2407.15337/) (2024)
- [RegGS: Unposed Sparse Views Gaussian Splatting with 3DGS Registration](https://mlsanity.com/paper/2507.08136/) (2025)
- [Cinematic Behavior Transfer via NeRF-based Differentiable Filming](https://mlsanity.com/paper/2311.17754/) (2023)
- [Neural Radiance Field-based Visual Rendering: A Comprehensive Review](https://mlsanity.com/paper/2404.00714/) (2024)
- [SfM-Free 3D Gaussian Splatting via Hierarchical Training](https://mlsanity.com/paper/2412.01553/) (2024)
- [Free360: Layered Gaussian Splatting for Unbounded 360-Degree View Synthesis from Extremely Sparse and Unposed Views](https://mlsanity.com/paper/2503.24382/) (2025)
- [Generic Objects as Pose Probes for Few-shot View Synthesis](https://mlsanity.com/paper/2408.16690/) (2024)
- [VideoLifter: Lifting Videos to 3D with Fast Hierarchical Stereo Alignment](https://mlsanity.com/paper/2501.01949/) (2025)
- [Sparfels: Fast Reconstruction from Sparse Unposed Imagery](https://mlsanity.com/paper/2505.02178/) (2025)
- [KFC-W: Generating 3D-Consistent Videos from Unposed Internet Photos](https://mlsanity.com/paper/2411.13549/) (2024)
- [BASED: Bundle-Adjusting Surgical Endoscopic Dynamic Video Reconstruction using Neural Radiance Fields](https://mlsanity.com/paper/2309.15329/) (2023)
- [CT-NeRF: Incremental Optimizing Neural Radiance Field and Poses with Complex Trajectory](https://mlsanity.com/paper/2404.13896/) (2024)
- [CAD-NeRF: Learning NeRFs from Uncalibrated Few-view Images by CAD Model Retrieval](https://mlsanity.com/paper/2411.02979/) (2024)
- [GBR: Generative Bundle Refinement for High-fidelity Gaussian Splatting with Enhanced Mesh Reconstruction](https://mlsanity.com/paper/2412.05908/) (2024)
- [TD-NeRF: Novel Truncated Depth Prior for Joint Camera Pose and Neural Radiance Field Optimization](https://mlsanity.com/paper/2405.07027/) (2024)
- [Disentangled Generation and Aggregation for Robust Radiance Fields](https://mlsanity.com/paper/2409.15715/) (2024)
- [NeRFs are Mirror Detectors: Using Structural Similarity for Multi-View Mirror Scene Reconstruction with 3D Surface Primitives](https://mlsanity.com/paper/2501.04074/) (2025)
- [Erasing the Ephemeral: Joint Camera Refinement and Transient Object Removal for Street View Synthesis](https://mlsanity.com/paper/2311.17634/) (2023)
- [Joint attitude estimation and 3D neural reconstruction of non-cooperative space objects](https://mlsanity.com/paper/2506.20638/) (2025)
- [NoPe-NeRF++: Local-to-Global Optimization of NeRF with No Pose Prior](https://mlsanity.com/paper/2511.17322/) (2025)
- [Robust SG-NeRF: Robust Scene Graph Aided Neural Surface Reconstruction](https://mlsanity.com/paper/2411.13620/) (2024)
- [A self-supervised cyclic neural-analytic approach for novel view synthesis and 3D reconstruction](https://mlsanity.com/paper/2503.03543/) (2025)
- [FA-BARF: Frequency Adapted Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2503.12086/) (2025)
- [STAMICS: Splat, Track And Map with Integrated Consistency and Semantics for Dense RGB-D SLAM](https://mlsanity.com/paper/2503.21425/) (2025)

## Abstract (arXiv)

A critical obstacle preventing NeRF models from being deployed broadly in the wild is their reliance on accurate camera poses. Consequently, there is growing interest in extending NeRF models to jointly optimize camera poses and scene representation, which offers an alternative to off-the-shelf SfM pipelines which have well-understood failure modes. Existing approaches for unposed NeRF operate under limited assumptions, such as a prior pose distribution or coarse pose initialization, making them less effective in a general setting. In this work, we propose a novel approach, LU-NeRF, that jointly estimates camera poses and neural radiance fields with relaxed assumptions on pose configuration. Our approach operates in a local-to-global manner, where we first optimize over local subsets of the data, dubbed mini-scenes. LU-NeRF estimates local pose and geometry for this challenging few-shot task. The mini-scene poses are brought into a global reference frame through a robust pose synchronization step, where a final global optimization of pose and scene can be performed. We show our LU-NeRF pipeline outperforms prior attempts at unposed NeRF without making restrictive assumptions on the pose prior. This allows us to operate in the general SE(3) pose setting, unlike the baselines. Our results also indicate our model can be complementary to feature-based SfM pipelines as it compares favorably to COLMAP on low-texture and low-resolution images.