# Lightweight High-Speed Photography Built on Coded Exposure and Implicit Neural Representation of Videos

arXiv: https://arxiv.org/abs/2311.13134 · HTML: https://mlsanity.com/paper/2311.13134/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 63 references that are in the map.
- First posted on arXiv 22 Nov 2023.

## Builds on (most cited first, 30 of 32)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring](https://mlsanity.com/paper/1612.02177/) (2016)
- [Blind Image Quality Assessment Using A Deep Bilinear Convolutional Neural Network](https://mlsanity.com/paper/1907.02665/) (2019)
- [NeRV: Neural Representations for Videos](https://mlsanity.com/paper/2110.13903/) (2021)
- [Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models](https://mlsanity.com/paper/2208.06677/) (2022)
- [Bringing a Blurry Frame Alive at High Frame-Rate with an Event Camera](https://mlsanity.com/paper/1811.10180/) (2018)
- [VideoINR: Learning Video Implicit Neural Representation for Continuous Space-Time Super-Resolution](https://mlsanity.com/paper/2206.04647/) (2022)
- [HNeRV: A Hybrid Neural Representation for Videos](https://mlsanity.com/paper/2304.02633/) (2023)
- [Deep Image Deblurring: A Survey](https://mlsanity.com/paper/2201.10700/) (2022)
- [E-NeRV: Expedite Neural Video Representation with Disentangled Spatial-Temporal Context](https://mlsanity.com/paper/2207.08132/) (2022)
- [Coded aperture compressive temporal imaging](https://mlsanity.com/paper/1302.2575/) (2013)
- [Learning to Extract a Video Sequence from a Single Motion-Blurred Image](https://mlsanity.com/paper/1804.04065/) (2018)
- [Blurry Video Frame Interpolation](https://mlsanity.com/paper/2002.12259/) (2020)
- [Bringing Alive Blurred Moments](https://mlsanity.com/paper/1804.02913/) (2018)
- [RSTT: Real-time Spatial Temporal Transformer for Space-Time Video Super-Resolution](https://mlsanity.com/paper/2203.14186/) (2022)
- [Reblur2Deblur: Deblurring Videos via Self-Supervised Learning](https://mlsanity.com/paper/1801.05117/) (2018)
- [Clean Images are Hard to Reblur: Exploiting the Ill-Posed Inverse Task for Dynamic Scene Deblurring](https://mlsanity.com/paper/2104.12665/) (2021)
- [DeFMO: Deblurring and Shape Recovery of Fast Moving Objects](https://mlsanity.com/paper/2012.00595/) (2020)
- [Animation from Blur: Multi-modal Blur Decomposition with Motion Guidance](https://mlsanity.com/paper/2207.10123/) (2022)
- [SCI: A Spectrum Concentrated Implicit Neural Compression for Biomedical Data](https://mlsanity.com/paper/2209.15180/) (2022)
- [Learning Cross-Video Neural Representations for High-Quality Frame Interpolation](https://mlsanity.com/paper/2203.00137/) (2022)
- [Restoration of Video Frames from a Single Blurred Image with Motion Understanding](https://mlsanity.com/paper/2104.09134/) (2021)
- [INFWIDE: Image and Feature Space Wiener Deconvolution Network for Non-blind Image Deblurring in Low-Light Conditions](https://mlsanity.com/paper/2207.08201/) (2022)
- [Video Reconstruction from a Single Motion Blurred Image using Learned Dynamic Phase Coding](https://mlsanity.com/paper/2112.14768/) (2021)
- [A Unified Framework for Compressive Video Recovery from Coded Exposure Techniques](https://mlsanity.com/paper/2011.05532/) (2020)

## Abstract (arXiv)

The demand for compact cameras capable of recording high-speed scenes with high resolution is steadily increasing. However, achieving such capabilities often entails high bandwidth requirements, resulting in bulky, heavy systems unsuitable for low-capacity platforms. To address this challenge, leveraging a coded exposure setup to encode a frame sequence into a blurry snapshot and subsequently retrieve the latent sharp video presents a lightweight solution. Nevertheless, restoring motion from blur remains a formidable challenge due to the inherent ill-posedness of motion blur decomposition, the intrinsic ambiguity in motion direction, and the diverse motions present in natural videos. In this study, we propose a novel approach to address these challenges by combining the classical coded exposure imaging technique with the emerging implicit neural representation for videos. We strategically embed motion direction cues into the blurry image during the imaging process. Additionally, we develop a novel implicit neural representation based blur decomposition network to sequentially extract the latent video frames from the blurry image, leveraging the embedded motion direction cues. To validate the effectiveness and efficiency of our proposed framework, we conduct extensive experiments using benchmark datasets and real-captured blurry images. The results demonstrate that our approach significantly outperforms existing methods in terms of both quality and flexibility. The code for our work is available at .https://github.com/zhihongz/BDINR