# On-the-Fly SfM: What you capture is What you get

arXiv: https://arxiv.org/abs/2309.11883 · HTML: https://mlsanity.com/paper/2309.11883/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 48 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Builds on (most cited first, 15 of 15)

- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018)
- [Fine-tuning CNN Image Retrieval with No Human Annotation](https://mlsanity.com/paper/1711.02512/) (2017)
- [Back to the Feature: Learning Robust Camera Localization from Pixels to Pose](https://mlsanity.com/paper/2103.09213/) (2021)
- [On the Limits of Pseudo Ground Truth in Visual Camera Re-localisation](https://mlsanity.com/paper/2109.00524/) (2021)
- [A General Optimization-based Framework for Local Odometry Estimation with Multiple Sensors](https://mlsanity.com/paper/1901.03638/) (2019)
- [Baseline Desensitizing In Translation Averaging](https://mlsanity.com/paper/1901.00643/) (2019)
- [Matchable Image Retrieval by Learning from Surface Reconstruction](https://mlsanity.com/paper/1811.10343/) (2018)
- [Algebraic Characterization of Essential Matrices and Their Averaging in Multiview Settings](https://mlsanity.com/paper/1904.02663/) (2019)
- [MegBA: A GPU-Based Distributed Library for Large-Scale Bundle Adjustment](https://mlsanity.com/paper/2112.01349/) (2021)
- [Hierarchical structure-and-motion recovery from uncalibrated images](https://mlsanity.com/paper/1506.00395/) (2015)
- [RPBA -- Robust Parallel Bundle Adjustment Based on Covariance Information](https://mlsanity.com/paper/1910.08138/) (2019)

## Built on by (most cited first, 4 of 4)

- [Gaussian On-the-Fly Splatting: A Progressive Framework for Robust Near Real-Time 3DGS Optimization](https://mlsanity.com/paper/2503.13086/) (2025)
- [SfM on-the-fly: Get better 3D from What You Capture](https://mlsanity.com/paper/2407.03939/) (2024)
- [OF3GS: On-the-Fly Feed-Forward 3D Gaussian Splatting from Unposed Images](https://mlsanity.com/paper/2606.03254/) (2026)
- [Exploring the potential of collaborative UAV 3D mapping in Kenyan savanna for wildlife research](https://mlsanity.com/paper/2409.15914/) (2024)

## Abstract (arXiv)

Over the last decades, ample achievements have been made on Structure from motion (SfM). However, the vast majority of them basically work in an offline manner, i.e., images are firstly captured and then fed together into a SfM pipeline for obtaining poses and sparse point cloud. In this work, on the contrary, we present an on-the-fly SfM: running online SfM while image capturing, the newly taken On-the-Fly image is online estimated with the corresponding pose and points, i.e., what you capture is what you get. Specifically, our approach firstly employs a vocabulary tree that is unsupervised trained using learning-based global features for fast image retrieval of newly fly-in image. Then, a robust feature matching mechanism with least squares (LSM) is presented to improve image registration performance. Finally, via investigating the influence of newly fly-in image's connected neighboring images, an efficient hierarchical weighted local bundle adjustment (BA) is used for optimization. Extensive experimental results demonstrate that on-the-fly SfM can meet the goal of robustly registering the images while capturing in an online way.