# Self-Supervised Real-time Video Stabilization

arXiv: https://arxiv.org/abs/2111.05980 · HTML: https://mlsanity.com/paper/2111.05980/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 33 references that are in the map.
- First posted on arXiv 10 Nov 2021.

## Builds on (most cited first, 7 of 7)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [Deep Image Homography Estimation](https://mlsanity.com/paper/1606.03798/) (2016)
- [Unsupervised Deep Homography: A Fast and Robust Homography Estimation Model](https://mlsanity.com/paper/1709.03966/) (2017)
- [Deep Iterative Frame Interpolation for Full-frame Video Stabilization](https://mlsanity.com/paper/1909.02641/) (2019)
- [Hybrid Neural Fusion for Full-frame Video Stabilization](https://mlsanity.com/paper/2102.06205/) (2021)

## Built on by (most cited first, 4 of 4)

- [Harnessing Meta-Learning for Improving Full-Frame Video Stabilization](https://mlsanity.com/paper/2403.03662/) (2024)
- [Stable and Consistent Prediction of 3D Characteristic Orientation via Invariant Residual Learning](https://mlsanity.com/paper/2306.11406/) (2023)
- [Putting 3D Spatially Sparse Networks on a Diet](https://mlsanity.com/paper/2112.01316/) (2021)
- [Harnessing Meta-Learning for Controllable Full-Frame Video Stabilization](https://mlsanity.com/paper/2508.18859/) (2025)

## Abstract (arXiv)

Videos are a popular media form, where online video streaming has recently gathered much popularity. In this work, we propose a novel method of real-time video stabilization - transforming a shaky video to a stabilized video as if it were stabilized via gimbals in real-time. Our framework is trainable in a self-supervised manner, which does not require data captured with special hardware setups (i.e., two cameras on a stereo rig or additional motion sensors). Our framework consists of a transformation estimator between given frames for global stability adjustments, followed by scene parallax reduction module via spatially smoothed optical flow for further stability. Then, a margin inpainting module fills in the missing margin regions created during stabilization to reduce the amount of post-cropping. These sequential steps reduce distortion and margin cropping to a minimum while enhancing stability. Hence, our approach outperforms state-of-the-art real-time video stabilization methods as well as offline methods that require camera trajectory optimization. Our method procedure takes approximately 24.3 ms yielding 41 fps regardless of resolution (e.g., 480p or 1080p).