# Leveraging Synthetic Data to Learn Video Stabilization Under Adverse Conditions

arXiv: https://arxiv.org/abs/2208.12763 · HTML: https://mlsanity.com/paper/2208.12763/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 45 references that are in the map.
- First posted on arXiv 26 Aug 2022.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels](https://mlsanity.com/paper/1905.05040/) (2019)
- [R2D2: Repeatable and Reliable Detector and Descriptor](https://mlsanity.com/paper/1906.06195/) (2019)
- [Learning Two-View Correspondences and Geometry Using Order-Aware Network](https://mlsanity.com/paper/1908.04964/) (2019)
- [Deep Image Homography Estimation](https://mlsanity.com/paper/1606.03798/) (2016)
- [ASLFeat: Learning Local Features of Accurate Shape and Localization](https://mlsanity.com/paper/2003.10071/) (2020)
- [MAGSAC: marginalizing sample consensus](https://mlsanity.com/paper/1803.07469/) (2018)
- [GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds](https://mlsanity.com/paper/2104.07659/) (2021)
- [Content-Aware Unsupervised Deep Homography Estimation](https://mlsanity.com/paper/1909.05983/) (2019)
- [Motion Basis Learning for Unsupervised Deep Homography Estimation with Subspace Projection](https://mlsanity.com/paper/2103.15346/) (2021)
- [Deep Iterative Frame Interpolation for Full-frame Video Stabilization](https://mlsanity.com/paper/1909.02641/) (2019)
- [Hybrid Neural Fusion for Full-frame Video Stabilization](https://mlsanity.com/paper/2102.06205/) (2021)
- [Play and Learn: Using Video Games to Train Computer Vision Models](https://mlsanity.com/paper/1608.01745/) (2016)
- [DUT: Learning Video Stabilization by Simply Watching Unstable Videos](https://mlsanity.com/paper/2011.14574/) (2020)
- [Cinematic-L1 Video Stabilization with a Log-Homography Model](https://mlsanity.com/paper/2011.08144/) (2020)

## Built on by (most cited first, 1 of 1)

- [No Labels, No Look-Ahead: Unsupervised Online Video Stabilization with Classical Priors](https://mlsanity.com/paper/2602.23141/) (2026)

## Abstract (arXiv)

Video stabilization plays a central role to improve videos quality. However, despite the substantial progress made by these methods, they were, mainly, tested under standard weather and lighting conditions, and may perform poorly under adverse conditions. In this paper, we propose a synthetic-aware adverse weather robust algorithm for video stabilization that does not require real data and can be trained only on synthetic data. We also present Silver, a novel rendering engine to generate the required training data with an automatic ground-truth extraction procedure. Our approach uses our specially generated synthetic data for training an affine transformation matrix estimator avoiding the feature extraction issues faced by current methods. Additionally, since no video stabilization datasets under adverse conditions are available, we propose the novel VSAC105Real dataset for evaluation. We compare our method to five state-of-the-art video stabilization algorithms using two benchmarks. Our results show that current approaches perform poorly in at least one weather condition, and that, even training in a small dataset with synthetic data only, we achieve the best performance in terms of stability average score, distortion score, success rate, and average cropping ratio when considering all weather conditions. Hence, our video stabilization model generalizes well on real-world videos and does not require large-scale synthetic training data to converge.