# Video frame interpolation for high dynamic range sequences captured with dual-exposure sensors

arXiv: https://arxiv.org/abs/2206.09485 · HTML: https://mlsanity.com/paper/2206.09485/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 74 references that are in the map.
- First posted on arXiv 19 Jun 2022.

## Builds on (most cited first, 30 of 35)

- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Uformer: A General U-Shaped Transformer for Image Restoration](https://mlsanity.com/paper/2106.03106/) (2021)
- [Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring](https://mlsanity.com/paper/1612.02177/) (2016)
- [High Speed and High Dynamic Range Video with an Event Camera](https://mlsanity.com/paper/1906.07165/) (2019)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [Softmax Splatting for Video Frame Interpolation](https://mlsanity.com/paper/2003.05534/) (2020)
- [Depth-Aware Video Frame Interpolation](https://mlsanity.com/paper/1904.00830/) (2019)
- [Video Frame Interpolation via Adaptive Separable Convolution](https://mlsanity.com/paper/1708.01692/) (2017)
- [FILM: Frame Interpolation for Large Motion](https://mlsanity.com/paper/2202.04901/) (2022)
- [Quadratic video interpolation](https://mlsanity.com/paper/1911.00627/) (2019)
- [AdaCoF: Adaptive Collaboration of Flows for Video Frame Interpolation](https://mlsanity.com/paper/1907.10244/) (2019)
- [XVFI: eXtreme Video Frame Interpolation](https://mlsanity.com/paper/2103.16206/) (2021)
- [What Matters in Unsupervised Optical Flow](https://mlsanity.com/paper/2006.04902/) (2020)
- [BMBC:Bilateral Motion Estimation with Bilateral Cost Volume for Video Interpolation](https://mlsanity.com/paper/2007.12622/) (2020)
- [Asymmetric Bilateral Motion Estimation for Video Frame Interpolation](https://mlsanity.com/paper/2108.06815/) (2021)
- [Deep Image Deblurring: A Survey](https://mlsanity.com/paper/2201.10700/) (2022)
- [From Motion Blur to Motion Flow: a Deep Learning Solution for Removing Heterogeneous Motion Blur](https://mlsanity.com/paper/1612.02583/) (2016)
- [Blurry Video Frame Interpolation](https://mlsanity.com/paper/2002.12259/) (2020)
- [HDR Video Reconstruction: A Coarse-to-fine Network and A Real-world Benchmark Dataset](https://mlsanity.com/paper/2103.14943/) (2021)
- [Deep Learning for HDR Imaging: State-of-the-Art and Future Trends](https://mlsanity.com/paper/2110.10394/) (2021)
- [Unsupervised Video Interpolation Using Cycle Consistency](https://mlsanity.com/paper/1906.05928/) (2019)
- [All at Once: Temporally Adaptive Multi-Frame Interpolation with Advanced Motion Modeling](https://mlsanity.com/paper/2007.11762/) (2020)
- [Video Frame Interpolation without Temporal Priors](https://mlsanity.com/paper/2112.01161/) (2021)
- [Convolutional Sparse Coding for High Dynamic Range Imaging](https://mlsanity.com/paper/1806.04942/) (2018)
- [ALANET: Adaptive Latent Attention Network forJoint Video Deblurring and Interpolation](https://mlsanity.com/paper/2009.01005/) (2020)
- [HDR Video Reconstruction with Tri-Exposure Quad-Bayer Sensors](https://mlsanity.com/paper/2103.10982/) (2021)

## Built on by (most cited first, 1 of 1)

- [Learning Images Across Scales Using Adversarial Training](https://mlsanity.com/paper/2406.08924/) (2024)

## Abstract (arXiv)

Video frame interpolation (VFI) enables many important applications that might involve the temporal domain, such as slow motion playback, or the spatial domain, such as stop motion sequences. We are focusing on the former task, where one of the key challenges is handling high dynamic range (HDR) scenes in the presence of complex motion. To this end, we explore possible advantages of dual-exposure sensors that readily provide sharp short and blurry long exposures that are spatially registered and whose ends are temporally aligned. This way, motion blur registers temporally continuous information on the scene motion that, combined with the sharp reference, enables more precise motion sampling within a single camera shot. We demonstrate that this facilitates a more complex motion reconstruction in the VFI task, as well as HDR frame reconstruction that so far has been considered only for the originally captured frames, not in-between interpolated frames. We design a neural network trained in these tasks that clearly outperforms existing solutions. We also propose a metric for scene motion complexity that provides important insights into the performance of VFI methods at the test time.