# Context-Aware Video Reconstruction for Rolling Shutter Cameras

arXiv: https://arxiv.org/abs/2205.12912 · HTML: https://mlsanity.com/paper/2205.12912/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 68 references that are in the map.
- First posted on arXiv 25 May 2022.

## Builds on (most cited first, 30 of 40)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [Softmax Splatting for Video Frame Interpolation](https://mlsanity.com/paper/2003.05534/) (2020)
- [Depth-Aware Video Frame Interpolation](https://mlsanity.com/paper/1904.00830/) (2019)
- [Video Frame Synthesis using Deep Voxel Flow](https://mlsanity.com/paper/1702.02463/) (2017)
- [Video Frame Interpolation via Adaptive Convolution](https://mlsanity.com/paper/1703.07514/) (2017)
- [Context-aware Synthesis for Video Frame Interpolation](https://mlsanity.com/paper/1803.10967/) (2018)
- [Quadratic video interpolation](https://mlsanity.com/paper/1911.00627/) (2019)
- [What Matters in Unsupervised Optical Flow](https://mlsanity.com/paper/2006.04902/) (2020)
- [BMBC:Bilateral Motion Estimation with Bilateral Cost Volume for Video Interpolation](https://mlsanity.com/paper/2007.12622/) (2020)
- [MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement](https://mlsanity.com/paper/1810.08768/) (2018)
- [Occlusion Aware Unsupervised Learning of Optical Flow](https://mlsanity.com/paper/1711.05890/) (2017)
- [Asymmetric Bilateral Motion Estimation for Video Frame Interpolation](https://mlsanity.com/paper/2108.06815/) (2021)
- [CCVS: Context-aware Controllable Video Synthesis](https://mlsanity.com/paper/2107.08037/) (2021)
- [Deep Animation Video Interpolation in the Wild](https://mlsanity.com/paper/2104.02495/) (2021)
- [PhaseNet for Video Frame Interpolation](https://mlsanity.com/paper/1804.00884/) (2018)
- [Enhanced Quadratic Video Interpolation](https://mlsanity.com/paper/2009.04642/) (2020)
- [Unsupervised Video Interpolation Using Cycle Consistency](https://mlsanity.com/paper/1906.05928/) (2019)
- [All at Once: Temporally Adaptive Multi-Frame Interpolation with Advanced Motion Modeling](https://mlsanity.com/paper/2007.11762/) (2020)
- [Rolling-Shutter-Aware Differential SfM and Image Rectification](https://mlsanity.com/paper/1903.03943/) (2019)
- [Towards Rolling Shutter Correction and Deblurring in Dynamic Scenes](https://mlsanity.com/paper/2104.01601/) (2021)

## Built on by (most cited first, 15 of 15)

- [Gaussian Splatting on the Move: Blur and Rolling Shutter Compensation for Natural Camera Motion](https://mlsanity.com/paper/2403.13327/) (2024)
- [USB-NeRF: Unrolling Shutter Bundle Adjusted Neural Radiance Fields](https://mlsanity.com/paper/2310.02687/) (2023)
- [UniINR: Event-guided Unified Rolling Shutter Correction, Deblurring, and Interpolation](https://mlsanity.com/paper/2305.15078/) (2023)
- [Towards Nonlinear-Motion-Aware and Occlusion-Robust Rolling Shutter Correction](https://mlsanity.com/paper/2303.18125/) (2023)
- [RS-NeRF: Neural Radiance Fields from Rolling Shutter Images](https://mlsanity.com/paper/2407.10267/) (2024)
- [Self-supervised Learning to Bring Dual Reversed Rolling Shutter Images Alive](https://mlsanity.com/paper/2305.19862/) (2023)
- [Self-supervised Learning of Event-guided Video Frame Interpolation for Rolling Shutter Frames](https://mlsanity.com/paper/2306.15507/) (2023)
- [Neural Image Re-Exposure](https://mlsanity.com/paper/2305.13593/) (2023)
- [URS-NeRF: Unordered Rolling Shutter Bundle Adjustment for Neural Radiance Fields](https://mlsanity.com/paper/2403.10119/) (2024)
- [Motion Blur Decomposition with Cross-shutter Guidance](https://mlsanity.com/paper/2404.01120/) (2024)
- [Self-Supervised Scene Dynamic Recovery from Rolling Shutter Images and Events](https://mlsanity.com/paper/2304.06930/) (2023)
- [Moment-Reenacting: Inverse Motion Degradation with Cross-shutter Guidance](https://mlsanity.com/paper/2605.22423/) (2026)
- [SelfDRSC++: Self-Supervised Dual Reversed Rolling Shutter Correction via Video Interpolation](https://mlsanity.com/paper/2408.11411/) (2024)
- [Scanline-Aware Animatable Gaussian Avatars from Rolling-Shutter Videos](https://mlsanity.com/paper/2608.17314/) (2026)
- [HaRP: High Dynamic Range Photosequencing through Dual Reversed Shutter Scanning](https://mlsanity.com/paper/2609.28439/) (2026)

## Abstract (arXiv)

With the ubiquity of rolling shutter (RS) cameras, it is becoming increasingly attractive to recover the latent global shutter (GS) video from two consecutive RS frames, which also places a higher demand on realism. Existing solutions, using deep neural networks or optimization, achieve promising performance. However, these methods generate intermediate GS frames through image warping based on the RS model, which inevitably result in black holes and noticeable motion artifacts. In this paper, we alleviate these issues by proposing a context-aware GS video reconstruction architecture. It facilitates the advantages such as occlusion reasoning, motion compensation, and temporal abstraction. Specifically, we first estimate the bilateral motion field so that the pixels of the two RS frames are warped to a common GS frame accordingly. Then, a refinement scheme is proposed to guide the GS frame synthesis along with bilateral occlusion masks to produce high-fidelity GS video frames at arbitrary times. Furthermore, we derive an approximated bilateral motion field model, which can serve as an alternative to provide a simple but effective GS frame initialization for related tasks. Experiments on synthetic and real data show that our approach achieves superior performance over state-of-the-art methods in terms of objective metrics and subjective visual quality. Code is available at urlhttps://github.com/GitCVfb/CVR.