# Robust Frame-to-Frame Camera Rotation Estimation in Crowded Scenes

arXiv: https://arxiv.org/abs/2309.08588 · HTML: https://mlsanity.com/paper/2309.08588/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 78 references that are in the map.
- First posted on arXiv 15 Sep 2023.

## Builds on (most cited first, 10 of 10)

- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [It's Moving! A Probabilistic Model for Causal Motion Segmentation in Moving Camera Videos](https://mlsanity.com/paper/1604.00136/) (2016)
- [DiffPoseNet: Direct Differentiable Camera Pose Estimation](https://mlsanity.com/paper/2203.11174/) (2022)
- [An Efficient Solution to Non-Minimal Case Essential Matrix Estimation](https://mlsanity.com/paper/1903.09067/) (2019)
- [Globally Optimal Relative Pose Estimation with Gravity Prior](https://mlsanity.com/paper/2012.00458/) (2020)
- [Joint direct estimation of 3D geometry and 3D motion using spatio temporal gradients](https://mlsanity.com/paper/1805.06641/) (2018)
- [Fast and Robust Certifiable Estimation of the Relative Pose Between Two Calibrated Cameras](https://mlsanity.com/paper/2101.08524/) (2021)
- [Relative Pose Estimation of Calibrated Cameras with Known SE(3) Invariants](https://mlsanity.com/paper/2007.07686/) (2020)
- [RAUM-VO: Rotational Adjusted Unsupervised Monocular Visual Odometry](https://mlsanity.com/paper/2203.07162/) (2022)

## Built on by (most cited first, 3 of 3)

- [SC-NeRF: NeRF-based Point Cloud Reconstruction using a Stationary Camera for Agricultural Applications](https://mlsanity.com/paper/2503.21958/) (2025)
- [FLIGHT: Fibonacci Lattice-based Inference for Geometric Heading in real-Time](https://mlsanity.com/paper/2602.23115/) (2026)
- [Cross-View Yaw Estimation in Location Uncertainty with Line-Aligning Yaw Scoring](https://mlsanity.com/paper/2606.22094/) (2026)

## Abstract (arXiv)

We present an approach to estimating camera rotation in crowded, real-world scenes from handheld monocular video. While camera rotation estimation is a well-studied problem, no previous methods exhibit both high accuracy and acceptable speed in this setting. Because the setting is not addressed well by other datasets, we provide a new dataset and benchmark, with high-accuracy, rigorously verified ground truth, on 17 video sequences. Methods developed for wide baseline stereo (e.g., 5-point methods) perform poorly on monocular video. On the other hand, methods used in autonomous driving (e.g., SLAM) leverage specific sensor setups, specific motion models, or local optimization strategies (lagging batch processing) and do not generalize well to handheld video. Finally, for dynamic scenes, commonly used robustification techniques like RANSAC require large numbers of iterations, and become prohibitively slow. We introduce a novel generalization of the Hough transform on SO(3) to efficiently and robustly find the camera rotation most compatible with optical flow. Among comparably fast methods, ours reduces error by almost 50% over the next best, and is more accurate than any method, irrespective of speed. This represents a strong new performance point for crowded scenes, an important setting for computer vision. The code and the dataset are available at https://fabiendelattre.com/robust-rotation-estimation.