# Rolling Shutter Inversion: Bring Rolling Shutter Images to High Framerate Global Shutter Video

arXiv: https://arxiv.org/abs/2210.03040 · HTML: https://mlsanity.com/paper/2210.03040/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 64 references that are in the map.
- First posted on arXiv 6 Oct 2022.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [Softmax Splatting for Video Frame Interpolation](https://mlsanity.com/paper/2003.05534/) (2020)
- [Depth-Aware Video Frame Interpolation](https://mlsanity.com/paper/1904.00830/) (2019)
- [Video Frame Synthesis using Deep Voxel Flow](https://mlsanity.com/paper/1702.02463/) (2017)
- [Context-aware Synthesis for Video Frame Interpolation](https://mlsanity.com/paper/1803.10967/) (2018)
- [Quadratic video interpolation](https://mlsanity.com/paper/1911.00627/) (2019)
- [BMBC:Bilateral Motion Estimation with Bilateral Cost Volume for Video Interpolation](https://mlsanity.com/paper/2007.12622/) (2020)
- [Occlusion Aware Unsupervised Learning of Optical Flow](https://mlsanity.com/paper/1711.05890/) (2017)
- [PhaseNet for Video Frame Interpolation](https://mlsanity.com/paper/1804.00884/) (2018)
- [Blurry Video Frame Interpolation](https://mlsanity.com/paper/2002.12259/) (2020)
- [Enhanced Quadratic Video Interpolation](https://mlsanity.com/paper/2009.04642/) (2020)
- [Unsupervised Video Interpolation Using Cycle Consistency](https://mlsanity.com/paper/1906.05928/) (2019)
- [All at Once: Temporally Adaptive Multi-Frame Interpolation with Advanced Motion Modeling](https://mlsanity.com/paper/2007.11762/) (2020)
- [Rolling-Shutter-Aware Differential SfM and Image Rectification](https://mlsanity.com/paper/1903.03943/) (2019)
- [Towards Rolling Shutter Correction and Deblurring in Dynamic Scenes](https://mlsanity.com/paper/2104.01601/) (2021)
- [Rolling Shutter Camera Relative Pose: Generalized Epipolar Geometry](https://mlsanity.com/paper/1605.00475/) (2016)
- [From two rolling shutters to one global shutter](https://mlsanity.com/paper/2006.01964/) (2020)
- [SUNet: Symmetric Undistortion Network for Rolling Shutter Correction](https://mlsanity.com/paper/2108.04775/) (2021)
- [Self-Supervised Linear Motion Deblurring](https://mlsanity.com/paper/2002.04070/) (2020)
- [Image Stitching and Rectification for Hand-Held Cameras](https://mlsanity.com/paper/2008.09229/) (2020)
- [Relative Pose Estimation for Stereo Rolling Shutter Cameras](https://mlsanity.com/paper/2006.07807/) (2020)

## Built on by (most cited first, 6 of 6)

- [UniINR: Event-guided Unified Rolling Shutter Correction, Deblurring, and Interpolation](https://mlsanity.com/paper/2305.15078/) (2023)
- [Order-One Rolling Shutter Cameras](https://mlsanity.com/paper/2403.11295/) (2024)
- [Single-Scanline Relative Pose Estimation for Rolling Shutter Cameras](https://mlsanity.com/paper/2506.22069/) (2025)
- [Minimal Solvers for Full DoF Motion Estimation from Asynchronous Tracks](https://mlsanity.com/paper/2508.17537/) (2025)
- [Moment-Reenacting: Inverse Motion Degradation with Cross-shutter Guidance](https://mlsanity.com/paper/2605.22423/) (2026)
- [A Revisit of the Normalized Eight-Point Algorithm and A Self-Supervised Deep Solution](https://mlsanity.com/paper/2304.10771/) (2023)

## Abstract (arXiv)

A single rolling-shutter (RS) image may be viewed as a row-wise combination of a sequence of global-shutter (GS) images captured by a (virtual) moving GS camera within the exposure duration. Although RS cameras are widely used, the RS effect causes obvious image distortion especially in the presence of fast camera motion, hindering downstream computer vision tasks. In this paper, we propose to invert the RS image capture mechanism, i.e., recovering a continuous high framerate GS video from two time-consecutive RS frames. We call this task the RS temporal super-resolution (RSSR) problem. The RSSR is a very challenging task, and to our knowledge, no practical solution exists to date. This paper presents a novel deep-learning based solution. By leveraging the multi-view geometry relationship of the RS imaging process, our learning-based framework successfully achieves high framerate GS generation. Specifically, three novel contributions can be identified: (i) novel formulations for bidirectional RS undistortion flows under constant velocity as well as constant acceleration motion model. (ii) a simple linear scaling operation, which bridges the RS undistortion flow and regular optical flow. (iii) a new mutual conversion scheme between varying RS undistortion flows that correspond to different scanlines. Our method also exploits the underlying spatial-temporal geometric relationships within a deep learning framework, where no additional supervision is required beyond the necessary middle-scanline GS image. Building upon these contributions, we represent the very first rolling-shutter temporal super-resolution deep-network that is able to recover high framerate GS videos from just two RS frames. Extensive experimental results on both synthetic and real data show that our proposed method can produce high-quality GS image sequences with rich details, outperforming the state-of-the-art methods.