# Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy

arXiv: https://arxiv.org/abs/2009.08283 · HTML: https://mlsanity.com/paper/2009.08283/

## Key facts

- Cited by 125 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2020.

## Built on by (most cited first, 30 of 125)

- [E-RAFT: Dense Optical Flow from Event Cameras](https://mlsanity.com/paper/2108.10552/) (2021)
- [TimeLens: Event-based Video Frame Interpolation](https://mlsanity.com/paper/2106.07286/) (2021)
- [Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks](https://mlsanity.com/paper/2106.01862/) (2021)
- [EventNeRF: Neural Radiance Fields from a Single Colour Event Camera](https://mlsanity.com/paper/2206.11896/) (2022)
- [Secrets of Event-Based Optical Flow](https://mlsanity.com/paper/2207.10022/) (2022)
- [Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarks](https://mlsanity.com/paper/2302.08890/) (2023)
- [VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehicles](https://mlsanity.com/paper/2111.12083/) (2021)
- [Recent Event Camera Innovations: A Survey](https://mlsanity.com/paper/2408.13627/) (2024)
- [Learning Dense and Continuous Optical Flow from an Event Camera](https://mlsanity.com/paper/2211.09078/) (2022)
- [Event-based Video Reconstruction via Potential-assisted Spiking Neural Network](https://mlsanity.com/paper/2201.10943/) (2022)
- [Ev-NeRF: Event Based Neural Radiance Field](https://mlsanity.com/paper/2206.12455/) (2022)
- [Unifying Motion Deblurring and Frame Interpolation with Events](https://mlsanity.com/paper/2203.12178/) (2022)
- [PRIOR: Prototype Representation Joint Learning from Medical Images and Reports](https://mlsanity.com/paper/2307.12577/) (2023)
- [EvDistill: Asynchronous Events to End-task Learning via Bidirectional Reconstruction-guided Cross-modal Knowledge Distillation](https://mlsanity.com/paper/2111.12341/) (2021)
- [Dense Continuous-Time Optical Flow from Events and Frames](https://mlsanity.com/paper/2203.13674/) (2022)
- [Deep Learning for HDR Imaging: State-of-the-Art and Future Trends](https://mlsanity.com/paper/2110.10394/) (2021)
- [Robust e-NeRF: NeRF from Sparse & Noisy Events under Non-Uniform Motion](https://mlsanity.com/paper/2309.08596/) (2023)
- [Masked Event Modeling: Self-Supervised Pretraining for Event Cameras](https://mlsanity.com/paper/2212.10368/) (2022)
- [Taming Contrast Maximization for Learning Sequential, Low-latency, Event-based Optical Flow](https://mlsanity.com/paper/2303.05214/) (2023)
- [HyperE2VID: Improving Event-Based Video Reconstruction via Hypernetworks](https://mlsanity.com/paper/2305.06382/) (2023)
- [EventBind: Learning a Unified Representation to Bind Them All for Event-based Open-world Understanding](https://mlsanity.com/paper/2308.03135/) (2023)
- [Seeing Motion at Nighttime with an Event Camera](https://mlsanity.com/paper/2404.11884/) (2024)
- [Optical flow estimation from event-based cameras and spiking neural networks](https://mlsanity.com/paper/2302.06492/) (2023)
- [Dual Transfer Learning for Event-based End-task Prediction via Pluggable Event to Image Translation](https://mlsanity.com/paper/2109.01801/) (2021)
- [TMA: Temporal Motion Aggregation for Event-based Optical Flow](https://mlsanity.com/paper/2303.11629/) (2023)
- [Lightweight Event-based Optical Flow Estimation via Iterative Deblurring](https://mlsanity.com/paper/2211.13726/) (2022)
- [Fully neuromorphic vision and control for autonomous drone flight](https://mlsanity.com/paper/2303.08778/) (2023)
- [Formulating Event-based Image Reconstruction as a Linear Inverse Problem with Deep Regularization using Optical Flow](https://mlsanity.com/paper/2112.06242/) (2021)
- [Label-Free Event-based Object Recognition via Joint Learning with Image Reconstruction from Events](https://mlsanity.com/paper/2308.09383/) (2023)
- [Real-Time Optical Flow for Vehicular Perception with Low- and High-Resolution Event Cameras](https://mlsanity.com/paper/2112.10591/) (2021)

## Abstract (arXiv)

Event cameras are novel vision sensors that sample, in an asynchronous fashion, brightness increments with low latency and high temporal resolution. The resulting streams of events are of high value by themselves, especially for high speed motion estimation. However, a growing body of work has also focused on the reconstruction of intensity frames from the events, as this allows bridging the gap with the existing literature on appearance- and frame-based computer vision. Recent work has mostly approached this problem using neural networks trained with synthetic, ground-truth data. In this work we approach, for the first time, the intensity reconstruction problem from a self-supervised learning perspective. Our method, which leverages the knowledge of the inner workings of event cameras, combines estimated optical flow and the event-based photometric constancy to train neural networks without the need for any ground-truth or synthetic data. Results across multiple datasets show that the performance of the proposed self-supervised approach is in line with the state-of-the-art. Additionally, we propose a novel, lightweight neural network for optical flow estimation that achieves high speed inference with only a minor drop in performance.