# Learning Optical Flow from Event Camera with Rendered Dataset

arXiv: https://arxiv.org/abs/2303.11011 · HTML: https://mlsanity.com/paper/2303.11011/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 60 references that are in the map.
- First posted on arXiv 20 Mar 2023.

## Builds on (most cited first, 30 of 38)

- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [GMFlow: Learning Optical Flow via Global Matching](https://mlsanity.com/paper/2111.13680/) (2021)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion](https://mlsanity.com/paper/1812.08156/) (2018)
- [DSEC: A Stereo Event Camera Dataset for Driving Scenarios](https://mlsanity.com/paper/2103.06011/) (2021)
- [FlowFormer: A Transformer Architecture for Optical Flow](https://mlsanity.com/paper/2203.16194/) (2022)
- [The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception](https://mlsanity.com/paper/1801.10202/) (2018)
- [EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras](https://mlsanity.com/paper/1802.06898/) (2018)
- [Learning to Estimate Hidden Motions with Global Motion Aggregation](https://mlsanity.com/paper/2104.02409/) (2021)
- [HATS: Histograms of Averaged Time Surfaces for Robust Event-based Object Classification](https://mlsanity.com/paper/1803.07913/) (2018)
- [End-to-End Learning of Representations for Asynchronous Event-Based Data](https://mlsanity.com/paper/1904.08245/) (2019)
- [Video to Events: Recycling Video Datasets for Event Cameras](https://mlsanity.com/paper/1912.03095/) (2019)
- [E-RAFT: Dense Optical Flow from Event Cameras](https://mlsanity.com/paper/2108.10552/) (2021)
- [Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation](https://mlsanity.com/paper/1904.05290/) (2019)
- [Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy](https://mlsanity.com/paper/2009.08283/) (2020)
- [Deep Flow-Guided Video Inpainting](https://mlsanity.com/paper/1905.02884/) (2019)
- [AutoFlow: Learning a Better Training Set for Optical Flow](https://mlsanity.com/paper/2104.14544/) (2021)
- [Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences](https://mlsanity.com/paper/1610.09513/) (2016)
- [Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks](https://mlsanity.com/paper/2106.01862/) (2021)
- [Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks](https://mlsanity.com/paper/2003.06696/) (2020)
- [Graph-Based Object Classification for Neuromorphic Vision Sensing](https://mlsanity.com/paper/1908.06648/) (2019)
- [DDD17: End-To-End DAVIS Driving Dataset](https://mlsanity.com/paper/1711.01458/) (2017)
- [CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flow](https://mlsanity.com/paper/2203.16896/) (2022)
- [Global Matching with Overlapping Attention for Optical Flow Estimation](https://mlsanity.com/paper/2203.11335/) (2022)
- [Learning Dense and Continuous Optical Flow from an Event Camera](https://mlsanity.com/paper/2211.09078/) (2022)
- [SKFlow: Learning Optical Flow with Super Kernels](https://mlsanity.com/paper/2205.14623/) (2022)
- [Spatio-Temporal Recurrent Networks for Event-Based Optical Flow Estimation](https://mlsanity.com/paper/2109.04871/) (2021)

## Built on by (most cited first, 26 of 26)

- [Recent Event Camera Innovations: A Survey](https://mlsanity.com/paper/2408.13627/) (2024)
- [GAFlow: Incorporating Gaussian Attention into Optical Flow](https://mlsanity.com/paper/2309.16217/) (2023)
- [EventHDR: from Event to High-Speed HDR Videos and Beyond](https://mlsanity.com/paper/2409.17029/) (2024)
- [Temporal Event Stereo via Joint Learning with Stereoscopic Flow](https://mlsanity.com/paper/2407.10831/) (2024)
- [Unsupervised Joint Learning of Optical Flow and Intensity with Event Cameras](https://mlsanity.com/paper/2503.17262/) (2025)
- [Accurate and Efficient Event-based Semantic Segmentation Using Adaptive Spiking Encoder-Decoder Network](https://mlsanity.com/paper/2304.11857/) (2023)
- [ADV2E: Bridging the Gap Between Analogue Circuit and Discrete Frames in the Video-to-Events Simulator](https://mlsanity.com/paper/2411.12250/) (2024)
- [EMatch: A Unified Framework for Event-based Optical Flow and Stereo Matching](https://mlsanity.com/paper/2407.21735/) (2024)
- [Learning Normal Flow Directly From Event Neighborhoods](https://mlsanity.com/paper/2412.11284/) (2024)
- [Vector-Symbolic Architecture for Event-Based Optical Flow](https://mlsanity.com/paper/2405.08300/) (2024)
- [ERetinex: Event Camera Meets Retinex Theory for Low-Light Image Enhancement](https://mlsanity.com/paper/2503.02484/) (2025)
- [V2V: Scaling Event-Based Vision through Efficient Video-to-Voxel Simulation](https://mlsanity.com/paper/2505.16797/) (2025)
- [Learning Efficient Meshflow and Optical Flow from Event Cameras](https://mlsanity.com/paper/2510.04111/) (2025)
- [Perturbed State Space Feature Encoders for Optical Flow with Event Cameras](https://mlsanity.com/paper/2504.10669/) (2025)
- [EvRWKV: A Continuous Interactive RWKV Framework for Effective Event-Guided Low-Light Image Enhancement](https://mlsanity.com/paper/2507.03184/) (2025)
- [E-MoFlow: Learning Egomotion and Optical Flow from Event Data via Implicit Regularization](https://mlsanity.com/paper/2510.12753/) (2025)
- [SDformerFlow: Spatiotemporal swin spikeformer for event-based optical flow estimation](https://mlsanity.com/paper/2409.04082/) (2024)
- [BAT: Learning Event-based Optical Flow with Bidirectional Adaptive Temporal Correlation](https://mlsanity.com/paper/2503.03256/) (2025)
- [Learning Flow-Guided Registration for RGB-Event Semantic Segmentation](https://mlsanity.com/paper/2505.01548/) (2025)
- [Uncertainty-Weighted Fusion of Image and Synthetic Event for Video Anomaly Detection](https://mlsanity.com/paper/2505.02393/) (2025)
- [DMAligner: Enhancing Image Alignment via Diffusion Model Based View Synthesis](https://mlsanity.com/paper/2602.23022/) (2026)
- [From Contrast to Consistency: Rethinking Event-based Continuous-Time Optical Flow Estimation](https://mlsanity.com/paper/2605.25570/) (2026)
- [eStonefish-Scenes: A Sim-to-Real Validated and Robot-Centric Event-based Optical Flow Dataset for Underwater Vehicles](https://mlsanity.com/paper/2505.13309/) (2025)
- [UEOF: A Benchmark Dataset for Underwater Event-Based Optical Flow](https://mlsanity.com/paper/2601.10054/) (2026)
- [Minimal Solvers for Full-DoF Motion Estimation from Asynchronous Differential SfM](https://mlsanity.com/paper/2606.09218/) (2026)
- [EVIS: Real-Time Event Camera Simulation with Multimodal Supervision in NVIDIA Isaac Sim](https://mlsanity.com/paper/2607.08098/) (2026)

## Abstract (arXiv)

We study the problem of estimating optical flow from event cameras. One important issue is how to build a high-quality event-flow dataset with accurate event values and flow labels. Previous datasets are created by either capturing real scenes by event cameras or synthesizing from images with pasted foreground objects. The former case can produce real event values but with calculated flow labels, which are sparse and inaccurate. The later case can generate dense flow labels but the interpolated events are prone to errors. In this work, we propose to render a physically correct event-flow dataset using computer graphics models. In particular, we first create indoor and outdoor 3D scenes by Blender with rich scene content variations. Second, diverse camera motions are included for the virtual capturing, producing images and accurate flow labels. Third, we render high-framerate videos between images for accurate events. The rendered dataset can adjust the density of events, based on which we further introduce an adaptive density module (ADM). Experiments show that our proposed dataset can facilitate event-flow learning, whereas previous approaches when trained on our dataset can improve their performances constantly by a relatively large margin. In addition, event-flow pipelines when equipped with our ADM can further improve performances.