# ROSEFusion: Random Optimization for Online Dense Reconstruction under Fast Camera Motion

arXiv: https://arxiv.org/abs/2105.05600 · HTML: https://mlsanity.com/paper/2105.05600/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 74 references that are in the map.
- First posted on arXiv 12 May 2021.

## Builds on (most cited first, 15 of 15)

- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [On-Manifold Preintegration for Real-Time Visual-Inertial Odometry](https://mlsanity.com/paper/1512.02363/) (2015)
- [PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Tracking](https://mlsanity.com/paper/1905.09304/) (2019)
- [DI-Fusion: Online Implicit 3D Reconstruction with Deep Priors](https://mlsanity.com/paper/2012.05551/) (2020)
- [Fusion-Aware Point Convolution for Online Semantic 3D Scene Segmentation](https://mlsanity.com/paper/2003.06233/) (2020)
- [RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction](https://mlsanity.com/paper/2011.14744/) (2020)
- [Visual-Inertial Odometry of Aerial Robots](https://mlsanity.com/paper/1906.03289/) (2019)
- [Real-Time Dense Mapping for Self-driving Vehicles using Fisheye Cameras](https://mlsanity.com/paper/1809.06132/) (2018)
- [Object-Aware Guidance for Autonomous Scene Reconstruction](https://mlsanity.com/paper/1805.07794/) (2018)
- [Distributed Bundle Adjustment](https://mlsanity.com/paper/1708.07954/) (2017)
- [A Controlled Particle Filter for Global Optimization](https://mlsanity.com/paper/1701.02413/) (2017)

## Built on by (most cited first, 2 of 2)

- [From Sim-to-Real: Toward General Event-based Low-light Frame Interpolation with Per-scene Optimization](https://mlsanity.com/paper/2406.08090/) (2024)
- [Large-displacement 3D Object Tracking with Hybrid Non-local Optimization](https://mlsanity.com/paper/2207.12620/) (2022)

## Abstract (arXiv)

Online reconstruction based on RGB-D sequences has thus far been restrained to relatively slow camera motions (<1m/s). Under very fast camera motion (e.g., 3m/s), the reconstruction can easily crumble even for the state-of-the-art methods. Fast motion brings two challenges to depth fusion: 1) the high nonlinearity of camera pose optimization due to large inter-frame rotations and 2) the lack of reliably trackable features due to motion blur. We propose to tackle the difficulties of fast-motion camera tracking in the absence of inertial measurements using random optimization, in particular, the Particle Filter Optimization (PFO). To surmount the computation-intensive particle sampling and update in standard PFO, we propose to accelerate the randomized search via updating a particle swarm template (PST). PST is a set of particles pre-sampled uniformly within the unit sphere in the 6D space of camera pose. Through moving and rescaling the pre-sampled PST guided by swarm intelligence, our method is able to drive tens of thousands of particles to locate and cover a good local optimum extremely fast and robustly. The particles, representing candidate poses, are evaluated with a fitness function defined based on depth-model conformance. Therefore, our method, being depth-only and correspondence-free, mitigates the motion blur impediment as ToF-based depths are often resilient to motion blur. Thanks to the efficient template-based particle set evolution and the effective fitness function, our method attains good quality pose tracking under fast camera motion (up to 4m/s) in a realtime framerate without including loop closure or global pose optimization. Through extensive evaluations on public datasets of RGB-D sequences, especially on a newly proposed benchmark of fast camera motion, we demonstrate the significant advantage of our method over the state of the arts.