# Self-supervised Learning of Event-guided Video Frame Interpolation for Rolling Shutter Frames

arXiv: https://arxiv.org/abs/2306.15507 · HTML: https://mlsanity.com/paper/2306.15507/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 75 references that are in the map.
- First posted on arXiv 27 Jun 2023.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [Softmax Splatting for Video Frame Interpolation](https://mlsanity.com/paper/2003.05534/) (2020)
- [End-to-End Learning of Representations for Asynchronous Event-Based Data](https://mlsanity.com/paper/1904.08245/) (2019)
- [Video to Events: Recycling Video Datasets for Event Cameras](https://mlsanity.com/paper/1912.03095/) (2019)
- [Context-aware Synthesis for Video Frame Interpolation](https://mlsanity.com/paper/1803.10967/) (2018)
- [E-RAFT: Dense Optical Flow from Event Cameras](https://mlsanity.com/paper/2108.10552/) (2021)
- [TimeLens: Event-based Video Frame Interpolation](https://mlsanity.com/paper/2106.07286/) (2021)
- [Quadratic video interpolation](https://mlsanity.com/paper/1911.00627/) (2019)
- [Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks](https://mlsanity.com/paper/1710.01992/) (2017)
- [BMBC:Bilateral Motion Estimation with Bilateral Cost Volume for Video Interpolation](https://mlsanity.com/paper/2007.12622/) (2020)
- [Time Lens++: Event-based Frame Interpolation with Parametric Non-linear Flow and Multi-scale Fusion](https://mlsanity.com/paper/2203.17191/) (2022)
- [Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarks](https://mlsanity.com/paper/2302.08890/) (2023)
- [PhaseNet for Video Frame Interpolation](https://mlsanity.com/paper/1804.00884/) (2018)
- [Event Based, Near Eye Gaze Tracking Beyond 10,000Hz](https://mlsanity.com/paper/2004.03577/) (2020)
- [All at Once: Temporally Adaptive Multi-Frame Interpolation with Advanced Motion Modeling](https://mlsanity.com/paper/2007.11762/) (2020)
- [EventSR: From Asynchronous Events to Image Reconstruction, Restoration, and Super-Resolution via End-to-End Adversarial Learning](https://mlsanity.com/paper/2003.07640/) (2020)
- [CED: Color Event Camera Dataset](https://mlsanity.com/paper/1904.10772/) (2019)
- [TimeReplayer: Unlocking the Potential of Event Cameras for Video Interpolation](https://mlsanity.com/paper/2203.13859/) (2022)
- [Learning Spatial-Temporal Implicit Neural Representations for Event-Guided Video Super-Resolution](https://mlsanity.com/paper/2303.13767/) (2023)
- [Real-Time Gaze Tracking with Event-Driven Eye Segmentation](https://mlsanity.com/paper/2201.07367/) (2022)
- [SUNet: Symmetric Undistortion Network for Rolling Shutter Correction](https://mlsanity.com/paper/2108.04775/) (2021)
- [Video Interpolation by Event-driven Anisotropic Adjustment of Optical Flow](https://mlsanity.com/paper/2208.09127/) (2022)
- [Rolling-Shutter Modelling for Direct Visual-Inertial Odometry](https://mlsanity.com/paper/1911.01015/) (2019)
- [Context-Aware Video Reconstruction for Rolling Shutter Cameras](https://mlsanity.com/paper/2205.12912/) (2022)
- [Deep Slow Motion Video Reconstruction with Hybrid Imaging System](https://mlsanity.com/paper/2002.12106/) (2020)
- [Learning Adaptive Warping for Real-World Rolling Shutter Correction](https://mlsanity.com/paper/2204.13886/) (2022)

## Built on by (most cited first, 5 of 5)

- [Event Camera Guided Visual Media Restoration & 3D Reconstruction: A Survey](https://mlsanity.com/paper/2509.09971/) (2025)
- [Can Video Diffusion Models Predict Past Frames? Bidirectional Cycle Consistency for Reversible Interpolation](https://mlsanity.com/paper/2604.01700/) (2026)
- [SelfDRSC++: Self-Supervised Dual Reversed Rolling Shutter Correction via Video Interpolation](https://mlsanity.com/paper/2408.11411/) (2024)
- [From Events to Enhancement: A Survey on Event-Based Imaging Technologies](https://mlsanity.com/paper/2505.05488/) (2025)
- [MiVID: Multi-Strategic Self-Supervision for Video Frame Interpolation using Diffusion Model](https://mlsanity.com/paper/2511.06019/) (2025)

## Abstract (arXiv)

Most consumer cameras use rolling shutter (RS) exposure, which often leads to distortions such as skew and jelly effects. These videos are further limited by bandwidth and frame rate constraints. In this paper, we explore the potential of event cameras, which offer high temporal resolution. We propose a framework to recover global shutter (GS) high-frame-rate videos without RS distortion by combining an RS camera and an event camera. Due to the lack of real-world datasets, our framework adopts a self-supervised strategy based on a displacement field, a dense 3D spatiotemporal representation of pixel motion during exposure. This enables mutual reconstruction between RS and GS frames and facilitates slow-motion recovery. We combine RS frames with the displacement field to generate GS frames, and integrate inverse mapping and RS frame warping for self-supervision. Experiments on four datasets show that our method removes distortion, reduces bandwidth usage by 94 percent, and achieves 16 ms per frame at 32x interpolation.