# E2V-SDE: From Asynchronous Events to Fast and Continuous Video Reconstruction via Neural Stochastic Differential Equations

arXiv: https://arxiv.org/abs/2206.07578 · HTML: https://mlsanity.com/paper/2206.07578/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 46 references that are in the map.
- First posted on arXiv 15 Jun 2022.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Neural Spline Flows](https://mlsanity.com/paper/1906.04032/) (2019)
- [Universal Differential Equations for Scientific Machine Learning](https://mlsanity.com/paper/2001.04385/) (2020)
- [High Speed and High Dynamic Range Video with an Event Camera](https://mlsanity.com/paper/1906.07165/) (2019)
- [Scalable Gradients for Stochastic Differential Equations](https://mlsanity.com/paper/2001.01328/) (2020)
- [The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception](https://mlsanity.com/paper/1801.10202/) (2018)
- [Learning Blind Video Temporal Consistency](https://mlsanity.com/paper/1808.00449/) (2018)
- [The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM](https://mlsanity.com/paper/1610.08336/) (2016)
- [Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit](https://mlsanity.com/paper/1905.09883/) (2019)
- [Neural SDEs as Infinite-Dimensional GANs](https://mlsanity.com/paper/2102.03657/) (2021)
- [Stochastic Normalizing Flows](https://mlsanity.com/paper/2002.06707/) (2020)
- [Neural Jump Stochastic Differential Equations](https://mlsanity.com/paper/1905.10403/) (2019)
- [Neural Rough Differential Equations for Long Time Series](https://mlsanity.com/paper/2009.08295/) (2020)
- [Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise](https://mlsanity.com/paper/1906.02355/) (2019)
- [Event-based High Dynamic Range Image and Very High Frame Rate Video Generation using Conditional Generative Adversarial Networks](https://mlsanity.com/paper/1811.08230/) (2018)
- [IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesis](https://mlsanity.com/paper/1807.06358/) (2018)
- [A Differentiable Programming System to Bridge Machine Learning and Scientific Computing](https://mlsanity.com/paper/1907.07587/) (2019)
- [Modeling Missing Data in Clinical Time Series with RNNs](https://mlsanity.com/paper/1606.04130/) (2016)
- [SDE-Net: Equipping Deep Neural Networks with Uncertainty Estimates](https://mlsanity.com/paper/2008.10546/) (2020)
- [NeuPDE: Neural Network Based Ordinary and Partial Differential Equations for Modeling Time-Dependent Data](https://mlsanity.com/paper/1908.03190/) (2019)
- [Progressive Temporal Feature Alignment Network for Video Inpainting](https://mlsanity.com/paper/2104.03507/) (2021)
- [Black-box Variational Inference for Stochastic Differential Equations](https://mlsanity.com/paper/1802.03335/) (2018)
- [Stochasticity in Neural ODEs: An Empirical Study](https://mlsanity.com/paper/2002.09779/) (2020)
- [Stochastic Differential Equations with Variational Wishart Diffusions](https://mlsanity.com/paper/2006.14895/) (2020)

## Abstract (arXiv)

Event cameras respond to brightness changes in the scene asynchronously and independently for every pixel. Due to the properties, these cameras have distinct features: high dynamic range (HDR), high temporal resolution, and low power consumption. However, the results of event cameras should be processed into an alternative representation for computer vision tasks. Also, they are usually noisy and cause poor performance in areas with few events. In recent years, numerous researchers have attempted to reconstruct videos from events. However, they do not provide good quality videos due to a lack of temporal information from irregular and discontinuous data. To overcome these difficulties, we introduce an E2V-SDE whose dynamics are governed in a latent space by Stochastic differential equations (SDE). Therefore, E2V-SDE can rapidly reconstruct images at arbitrary time steps and make realistic predictions on unseen data. In addition, we successfully adopted a variety of image composition techniques for improving image clarity and temporal consistency. By conducting extensive experiments on simulated and real-scene datasets, we verify that our model outperforms state-of-the-art approaches under various video reconstruction settings. In terms of image quality, the LPIPS score improves by up to 12% and the reconstruction speed is 87% higher than that of ET-Net.