# FrozenRecon: Pose-free 3D Scene Reconstruction with Frozen Depth Models

arXiv: https://arxiv.org/abs/2308.05733 · HTML: https://mlsanity.com/paper/2308.05733/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 50 references that are in the map.
- First posted on arXiv 10 Aug 2023.

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions](https://mlsanity.com/paper/1603.08182/) (2016)
- [DeepMVS: Learning Multi-view Stereopsis](https://mlsanity.com/paper/1804.00650/) (2018)
- [Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video](https://mlsanity.com/paper/1908.10553/) (2019)
- [NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video](https://mlsanity.com/paper/2104.00681/) (2021)
- [Consistent Video Depth Estimation](https://mlsanity.com/paper/2004.15021/) (2020)
- [Learning to Recover 3D Scene Shape from a Single Image](https://mlsanity.com/paper/2012.09365/) (2020)
- [Atlas: End-to-End 3D Scene Reconstruction from Posed Images](https://mlsanity.com/paper/2003.10432/) (2020)
- [Robust Consistent Video Depth Estimation](https://mlsanity.com/paper/2012.05901/) (2020)
- [DPSNet: End-to-end Deep Plane Sweep Stereo](https://mlsanity.com/paper/1905.00538/) (2019)
- [SC-DepthV3: Robust Self-supervised Monocular Depth Estimation for Dynamic Scenes](https://mlsanity.com/paper/2211.03660/) (2022)
- [Virtual Normal: Enforcing Geometric Constraints for Accurate and Robust Depth Prediction](https://mlsanity.com/paper/2103.04216/) (2021)
- [Towards Accurate Reconstruction of 3D Scene Shape from A Single Monocular Image](https://mlsanity.com/paper/2208.13241/) (2022)
- [Auto-Rectify Network for Unsupervised Indoor Depth Estimation](https://mlsanity.com/paper/2006.02708/) (2020)
- [PlanarRecon: Real-time 3D Plane Detection and Reconstruction from Posed Monocular Videos](https://mlsanity.com/paper/2206.07710/) (2022)
- [Towards 3D Scene Reconstruction from Locally Scale-Aligned Monocular Video Depth](https://mlsanity.com/paper/2202.01470/) (2022)

## Built on by (most cited first, 17 of 17)

- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass](https://mlsanity.com/paper/2501.13928/) (2025)
- [3D Reconstruction with Spatial Memory](https://mlsanity.com/paper/2408.16061/) (2024)
- [MUSt3R: Multi-view Network for Stereo 3D Reconstruction](https://mlsanity.com/paper/2503.01661/) (2025)
- [You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scale](https://mlsanity.com/paper/2412.06699/) (2024)
- [LONG3R: Long Sequence Streaming 3D Reconstruction](https://mlsanity.com/paper/2507.18255/) (2025)
- [DiffCalib: Reformulating Monocular Camera Calibration as Diffusion-Based Dense Incident Map Generation](https://mlsanity.com/paper/2405.15619/) (2024)
- [JOG3R: Towards 3D-Consistent Video Generators](https://mlsanity.com/paper/2501.01409/) (2025)
- [LaRI: Layered Ray Intersections for Single-view 3D Geometric Reasoning](https://mlsanity.com/paper/2504.18424/) (2025)
- [Open-Pose 3D Zero-Shot Learning: Benchmark and Challenges](https://mlsanity.com/paper/2312.07039/) (2023)
- [Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction-Reasoning Synergy](https://mlsanity.com/paper/2509.24385/) (2025)
- [UNO: Unified Self-Supervised Monocular Odometry for Platform-Agnostic Deployment](https://mlsanity.com/paper/2506.07013/) (2025)
- [RaCalNet: Radar Calibration Network for Sparse-Supervised Metric Depth Estimation](https://mlsanity.com/paper/2506.15560/) (2025)
- [StructSplat: Generalizable 3D Gaussian Splatting from Uncalibrated Sparse Views](https://mlsanity.com/paper/2606.28321/) (2026)
- [Improving Neural Indoor Surface Reconstruction with Mask-Guided Adaptive Consistency Constraints](https://mlsanity.com/paper/2309.09739/) (2023)
- [Enhancing Glass Surface Reconstruction via Depth Prior for Robot Navigation](https://mlsanity.com/paper/2604.18336/) (2026)
- [Unlocking the Power of Critical Factors for 3D Visual Geometry Estimation](https://mlsanity.com/paper/2604.21713/) (2026)

## Abstract (arXiv)

3D scene reconstruction is a long-standing vision task. Existing approaches can be categorized into geometry-based and learning-based methods. The former leverages multi-view geometry but can face catastrophic failures due to the reliance on accurate pixel correspondence across views. The latter was proffered to mitigate these issues by learning 2D or 3D representation directly. However, without a large-scale video or 3D training data, it can hardly generalize to diverse real-world scenarios due to the presence of tens of millions or even billions of optimization parameters in the deep network. Recently, robust monocular depth estimation models trained with large-scale datasets have been proven to possess weak 3D geometry prior, but they are insufficient for reconstruction due to the unknown camera parameters, the affine-invariant property, and inter-frame inconsistency. Here, we propose a novel test-time optimization approach that can transfer the robustness of affine-invariant depth models such as LeReS to challenging diverse scenes while ensuring inter-frame consistency, with only dozens of parameters to optimize per video frame. Specifically, our approach involves freezing the pre-trained affine-invariant depth model's depth predictions, rectifying them by optimizing the unknown scale-shift values with a geometric consistency alignment module, and employing the resulting scale-consistent depth maps to robustly obtain camera poses and achieve dense scene reconstruction, even in low-texture regions. Experiments show that our method achieves state-of-the-art cross-dataset reconstruction on five zero-shot testing datasets.