# Formulating Event-based Image Reconstruction as a Linear Inverse Problem with Deep Regularization using Optical Flow

arXiv: https://arxiv.org/abs/2112.06242 · HTML: https://mlsanity.com/paper/2112.06242/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 73 references that are in the map.
- First posted on arXiv 12 Dec 2021.

## Builds on (most cited first, 21 of 21)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Plug-and-Play Image Restoration with Deep Denoiser Prior](https://mlsanity.com/paper/2008.13751/) (2020)
- [Learning Deep CNN Denoiser Prior for Image Restoration](https://mlsanity.com/paper/1704.03264/) (2017)
- [High Speed and High Dynamic Range Video with an Event Camera](https://mlsanity.com/paper/1906.07165/) (2019)
- [Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades](https://mlsanity.com/paper/1507.07629/) (2015)
- [Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion](https://mlsanity.com/paper/1812.08156/) (2018)
- [DSEC: A Stereo Event Camera Dataset for Driving Scenarios](https://mlsanity.com/paper/2103.06011/) (2021)
- [EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras](https://mlsanity.com/paper/1802.06898/) (2018)
- [The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM](https://mlsanity.com/paper/1610.08336/) (2016)
- [A Unifying Contrast Maximization Framework for Event Cameras, with Applications to Motion, Depth, and Optical Flow Estimation](https://mlsanity.com/paper/1804.01306/) (2018)
- [E-RAFT: Dense Optical Flow from Event Cameras](https://mlsanity.com/paper/2108.10552/) (2021)
- [Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy](https://mlsanity.com/paper/2009.08283/) (2020)
- [Event-Based Motion Segmentation by Motion Compensation](https://mlsanity.com/paper/1904.01293/) (2019)
- [Continuous-time Intensity Estimation Using Event Cameras](https://mlsanity.com/paper/1811.00386/) (2018)
- [Event-based Motion Segmentation with Spatio-Temporal Graph Cuts](https://mlsanity.com/paper/2012.08730/) (2020)
- [Real-Time Intensity-Image Reconstruction for Event Cameras Using Manifold Regularisation](https://mlsanity.com/paper/1607.06283/) (2016)
- [EventSR: From Asynchronous Events to Image Reconstruction, Restoration, and Super-Resolution via End-to-End Adversarial Learning](https://mlsanity.com/paper/2003.07640/) (2020)
- [CED: Color Event Camera Dataset](https://mlsanity.com/paper/1904.10772/) (2019)
- [The Spatio-Temporal Poisson Point Process: A Simple Model for the Alignment of Event Camera Data](https://mlsanity.com/paper/2106.06887/) (2021)

## Built on by (most cited first, 28 of 28)

- [Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarks](https://mlsanity.com/paper/2302.08890/) (2023)
- [OpenESS: Event-based Semantic Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2405.05259/) (2024)
- [HyperE2VID: Improving Event-Based Video Reconstruction via Hypernetworks](https://mlsanity.com/paper/2305.06382/) (2023)
- [CMax-SLAM: Event-based Rotational-Motion Bundle Adjustment and SLAM System using Contrast Maximization](https://mlsanity.com/paper/2403.08119/) (2024)
- [EVREAL: Towards a Comprehensive Benchmark and Analysis Suite for Event-based Video Reconstruction](https://mlsanity.com/paper/2305.00434/) (2023)
- [Generalizing Event-Based Motion Deblurring in Real-World Scenarios](https://mlsanity.com/paper/2308.05932/) (2023)
- [EventAid: Benchmarking Event-aided Image/Video Enhancement Algorithms with Real-captured Hybrid Dataset](https://mlsanity.com/paper/2312.08220/) (2023)
- [ESVO2: Direct Visual-Inertial Odometry with Stereo Event Cameras](https://mlsanity.com/paper/2410.09374/) (2024)
- [EF-3DGS: Event-Aided Free-Trajectory 3D Gaussian Splatting](https://mlsanity.com/paper/2410.15392/) (2024)
- [Event-based Background-Oriented Schlieren](https://mlsanity.com/paper/2311.00434/) (2023)
- [EV-MGRFlowNet: Motion-Guided Recurrent Network for Unsupervised Event-based Optical Flow with Hybrid Motion-Compensation Loss](https://mlsanity.com/paper/2305.07853/) (2023)
- [Privacy-Preserving Visual Localization with Event Cameras](https://mlsanity.com/paper/2212.03177/) (2022)
- [LaSe-E2V: Towards Language-guided Semantic-Aware Event-to-Video Reconstruction](https://mlsanity.com/paper/2407.05547/) (2024)
- [Event-based Mosaicing Bundle Adjustment](https://mlsanity.com/paper/2409.07365/) (2024)
- [A Survey of 3D Reconstruction with Event Cameras](https://mlsanity.com/paper/2505.08438/) (2025)
- [Density Invariant Contrast Maximization for Neuromorphic Earth Observations](https://mlsanity.com/paper/2304.14125/) (2023)
- [Revisit Event Generation Model: Self-Supervised Learning of Event-to-Video Reconstruction with Implicit Neural Representations](https://mlsanity.com/paper/2407.18500/) (2024)
- [Event-based Photometric Bundle Adjustment](https://mlsanity.com/paper/2412.14111/) (2024)
- [Enhanced Event-Based Video Reconstruction with Motion Compensation](https://mlsanity.com/paper/2403.11961/) (2024)
- [Optimal OnTheFly Feedback Control of Event Sensors](https://mlsanity.com/paper/2408.12976/) (2024)
- [Dynamic Reconstruction from Neuromorphic Data](https://mlsanity.com/paper/2408.15465/) (2024)
- [Optical Linear Systems Framework for Event Sensing and Computational Neuromorphic Imaging](https://mlsanity.com/paper/2601.13498/) (2026)
- [Non-Uniform Exposure Imaging via Neuromorphic Shutter Control](https://mlsanity.com/paper/2404.13972/) (2024)
- [From Events to Enhancement: A Survey on Event-Based Imaging Technologies](https://mlsanity.com/paper/2505.05488/) (2025)
- [Paving the Way for Point Cloud Video Representation Learning Using A PDE Model](https://mlsanity.com/paper/2606.01604/) (2026)
- [Computation-Aware Event-to-Frame Reconstruction via Selective Attention](https://mlsanity.com/paper/2606.06142/) (2026)
- [ESAR: Event-Based Synthetic Aperture Reconstruction](https://mlsanity.com/paper/2607.15073/) (2026)
- [Modelling dynamic systems transfer functions from events in computational neuromorphic imaging](https://mlsanity.com/paper/2609.29863/) (2026)

## Abstract (arXiv)

Event cameras are novel bio-inspired sensors that measure per-pixel brightness differences asynchronously. Recovering brightness from events is appealing since the reconstructed images inherit the high dynamic range (HDR) and high-speed properties of events; hence they can be used in many robotic vision applications and to generate slow-motion HDR videos. However, state-of-the-art methods tackle this problem by training an event-to-image Recurrent Neural Network (RNN), which lacks explainability and is difficult to tune. In this work we show, for the first time, how tackling the combined problem of motion and brightness estimation leads us to formulate event-based image reconstruction as a linear inverse problem that can be solved without training an image reconstruction RNN. Instead, classical and learning-based regularizers are used to solve the problem and remove artifacts from the reconstructed images. The experiments show that the proposed approach generates images with visual quality on par with state-of-the-art methods despite only using data from a short time interval. State-of-the-art results are achieved using an image denoising Convolutional Neural Network (CNN) as the regularization function. The proposed regularized formulation and solvers have a unifying character because they can be applied also to reconstruct brightness from the second derivative. Additionally, the formulation is attractive because it can be naturally combined with super-resolution, motion-segmentation and color demosaicing. Code is available at https://github.com/tub-rip/event_based_image_rec_inverse_problem