# SceneFactory: A Workflow-centric and Unified Framework for Incremental Scene Modeling

arXiv: https://arxiv.org/abs/2405.07847 · HTML: https://mlsanity.com/paper/2405.07847/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 63 references that are in the map.
- First posted on arXiv 13 May 2024.

## Builds on (most cited first, 30 of 43)

- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021)
- [iMAP: Implicit Mapping and Positioning in Real-Time](https://mlsanity.com/paper/2103.12352/) (2021)
- [MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction](https://mlsanity.com/paper/2206.00665/) (2022)
- [Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching](https://mlsanity.com/paper/1912.06378/) (2019)
- [Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation](https://mlsanity.com/paper/2404.15506/) (2024)
- [Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image](https://mlsanity.com/paper/2307.10984/) (2023)
- [DeepMVS: Learning Multi-view Stereopsis](https://mlsanity.com/paper/1804.00650/) (2018)
- [DSAC - Differentiable RANSAC for Camera Localization](https://mlsanity.com/paper/1611.05705/) (2016)
- [Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference](https://mlsanity.com/paper/1902.10556/) (2019)
- [DeMoN: Depth and Motion Network for Learning Monocular Stereo](https://mlsanity.com/paper/1612.02401/) (2016)
- [Deep Patch Visual Odometry](https://mlsanity.com/paper/2208.04726/) (2022)
- [NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fields](https://mlsanity.com/paper/2210.13641/) (2022)
- [DeepV2D: Video to Depth with Differentiable Structure from Motion](https://mlsanity.com/paper/1812.04605/) (2018)
- [DKM: Dense Kernelized Feature Matching for Geometry Estimation](https://mlsanity.com/paper/2202.00667/) (2022)
- [ESLAM: Efficient Dense SLAM System Based on Hybrid Representation of Signed Distance Fields](https://mlsanity.com/paper/2211.11704/) (2022)
- [Cost Volume Pyramid Based Depth Inference for Multi-View Stereo](https://mlsanity.com/paper/1912.08329/) (2019)
- [CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM](https://mlsanity.com/paper/1804.00874/) (2018)
- [Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction](https://mlsanity.com/paper/1706.07036/) (2017)
- [DeepFactors: Real-Time Probabilistic Dense Monocular SLAM](https://mlsanity.com/paper/2001.05049/) (2020)
- [DeepTAM: Deep Tracking and Mapping](https://mlsanity.com/paper/1808.01900/) (2018)
- [Orbeez-SLAM: A Real-time Monocular Visual SLAM with ORB Features and NeRF-realized Mapping](https://mlsanity.com/paper/2209.13274/) (2022)

## Built on by (most cited first, 3 of 3)

- [HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Video](https://mlsanity.com/paper/2510.05560/) (2025)
- [SLAM-Former: Putting SLAM into One Transformer](https://mlsanity.com/paper/2509.16909/) (2025)
- [SLAMFormer-∞: Infinite SLAM Transformer for Unbounded Frontend and Backend Processing](https://mlsanity.com/paper/2608.03429/) (2026)

## Abstract (arXiv)

We present SceneFactory, a workflow-centric and unified framework for incremental scene modeling, that conveniently supports a wide range of applications, such as (unposed and/or uncalibrated) multi-view depth estimation, LiDAR completion, (dense) RGB-D/RGB-L/Mono/Depth-only reconstruction and SLAM. The workflow-centric design uses multiple blocks as the basis for constructing different production lines. The supported applications, i.e., productions avoid redundancy in their designs. Thus, the focus is placed on each block itself for independent expansion. To support all input combinations, our implementation consists of four building blocks that form SceneFactory: (1) tracking, (2) flexion, (3) depth estimation, and (4) scene reconstruction. The tracking block is based on Mono SLAM and is extended to support RGB-D and RGB-LiDAR (RGB-L) inputs. Flexion is used to convert the depth image (untrackable) into a trackable image. For general-purpose depth estimation, we propose an unposed & uncalibrated multi-view depth estimation model (U2-MVD) to estimate dense geometry. U2-MVD exploits dense bundle adjustment to solve for poses, intrinsics, and inverse depth. A semantic-aware ScaleCov step is then introduced to complete the multi-view depth. Relying on U2-MVD, SceneFactory both supports user-friendly 3D creation (with just images) and bridges the applications of Dense RGB-D and Dense Mono. For high-quality surface and color reconstruction, we propose Dual-purpose Multi-resolutional Neural Points (DM-NPs) for the first surface accessible Surface Color Field design, where we introduce Improved Point Rasterization (IPR) for point cloud based surface query. ...