# Incremental Joint Learning of Depth, Pose and Implicit Scene Representation on Monocular Camera in Large-scale Scenes

arXiv: https://arxiv.org/abs/2404.06050 · HTML: https://mlsanity.com/paper/2404.06050/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 89 references that are in the map.
- First posted on arXiv 9 Apr 2024.

## Builds on (most cited first, 30 of 39)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023)
- [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021)
- [INeRF: Inverting Neural Radiance Fields for Pose Estimation](https://mlsanity.com/paper/2012.05877/) (2020)
- [Neural RGB-D Surface Reconstruction](https://mlsanity.com/paper/2104.04532/) (2021)
- [NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior](https://mlsanity.com/paper/2212.07388/) (2022)
- [Self-Calibrating Neural Radiance Fields](https://mlsanity.com/paper/2108.13826/) (2021)
- [Back to the Feature: Learning Robust Camera Localization from Pixels to Pose](https://mlsanity.com/paper/2103.09213/) (2021)
- [NeRF--: Neural Radiance Fields Without Known Camera Parameters](https://mlsanity.com/paper/2102.07064/) (2021)
- [Progressively Optimized Local Radiance Fields for Robust View Synthesis](https://mlsanity.com/paper/2303.13791/) (2023)
- [SGS-SLAM: Semantic Gaussian Splatting For Neural Dense SLAM](https://mlsanity.com/paper/2402.03246/) (2024)
- [Unsupervised Scale-consistent Depth Learning from Video](https://mlsanity.com/paper/2105.11610/) (2021)
- [SNI-SLAM: Semantic Neural Implicit SLAM](https://mlsanity.com/paper/2311.11016/) (2023)
- [PLGSLAM: Progressive Neural Scene Represenation with Local to Global Bundle Adjustment](https://mlsanity.com/paper/2312.09866/) (2023)
- [Learning Single Camera Depth Estimation using Dual-Pixels](https://mlsanity.com/paper/1904.05822/) (2019)
- [What Makes Good Synthetic Training Data for Learning Disparity and Optical Flow Estimation?](https://mlsanity.com/paper/1801.06397/) (2018)
- [VolumeFusion: Deep Depth Fusion for 3D Scene Reconstruction](https://mlsanity.com/paper/2108.08623/) (2021)
- [SemGauss-SLAM: Dense Semantic Gaussian Splatting SLAM](https://mlsanity.com/paper/2403.07494/) (2024)
- [RegFormer: An Efficient Projection-Aware Transformer Network for Large-Scale Point Cloud Registration](https://mlsanity.com/paper/2303.12384/) (2023)
- [NeSLAM: Neural Implicit Mapping and Self-Supervised Feature Tracking With Depth Completion and Denoising](https://mlsanity.com/paper/2403.20034/) (2024)
- [Bridging the Domain Gap: Self-Supervised 3D Scene Understanding with Foundation Models](https://mlsanity.com/paper/2305.08776/) (2023)

## Built on by (most cited first, 15 of 15)

- [Time Series Modeling for Heart Rate Prediction: From ARIMA to Transformers](https://mlsanity.com/paper/2406.12199/) (2024)
- [Regional Style and Color Transfer](https://mlsanity.com/paper/2404.13880/) (2024)
- [What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Models](https://mlsanity.com/paper/2512.03422/) (2025)
- [UAVScenes: A Multi-Modal Dataset for UAVs](https://mlsanity.com/paper/2507.22412/) (2025)
- [Harnessing Earnings Reports for Stock Predictions: A QLoRA-Enhanced LLM Approach](https://mlsanity.com/paper/2408.06634/) (2024)
- [AR Overlay: Training Image Pose Estimation on Curved Surface in a Synthetic Way](https://mlsanity.com/paper/2409.14577/) (2024)
- [VPGS-SLAM: Voxel-based Progressive 3D Gaussian SLAM in Large-Scale Scenes](https://mlsanity.com/paper/2505.18992/) (2025)
- [Autonomous Navigation of Unmanned Vehicle Through Deep Reinforcement Learning](https://mlsanity.com/paper/2407.18962/) (2024)
- [Multi-Modal UAV Detection, Classification and Tracking Algorithm -- Technical Report for CVPR 2024 UG2 Challenge](https://mlsanity.com/paper/2405.16464/) (2024)
- [UniPR-3D: Towards Universal Visual Place Recognition with Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2512.21078/) (2025)
- [Exploiting Diffusion Prior for Out-of-Distribution Detection](https://mlsanity.com/paper/2406.11105/) (2024)
- [Reloc-VGGT: Visual Re-localization with Geometry Grounded Transformer](https://mlsanity.com/paper/2512.21883/) (2025)
- [UNO: Unified Self-Supervised Monocular Odometry for Platform-Agnostic Deployment](https://mlsanity.com/paper/2506.07013/) (2025)
- [An Efficient LiDAR-Camera Fusion Network for Multi-Class 3D Dynamic Object Detection and Trajectory Prediction](https://mlsanity.com/paper/2504.13647/) (2025)
- [Multi-Submap Implicit Neural SLAM with Local-to-Global Loop Closure for Large-Scale Scene Reconstruction](https://mlsanity.com/paper/2608.09146/) (2026)

## Abstract (arXiv)

Dense scene reconstruction for photo-realistic view synthesis has various applications, such as VR/AR, autonomous vehicles. However, most existing methods have difficulties in large-scale scenes due to three core challenges: textit(a) inaccurate depth input. Accurate depth input is impossible to get in real-world large-scale scenes. textit(b) inaccurate pose estimation. Most existing approaches rely on accurate pre-estimated camera poses. textit(c) insufficient scene representation capability. A single global radiance field lacks the capacity to effectively scale to large-scale scenes. To this end, we propose an incremental joint learning framework, which can achieve accurate depth, pose estimation, and large-scale scene reconstruction. A vision transformer-based network is adopted as the backbone to enhance performance in scale information estimation. For pose estimation, a feature-metric bundle adjustment (FBA) method is designed for accurate and robust camera tracking in large-scale scenes. In terms of implicit scene representation, we propose an incremental scene representation method to construct the entire large-scale scene as multiple local radiance fields to enhance the scalability of 3D scene representation. Extended experiments have been conducted to demonstrate the effectiveness and accuracy of our method in depth estimation, pose estimation, and large-scale scene reconstruction.