# Event-based Video Reconstruction via Potential-assisted Spiking Neural Network

arXiv: https://arxiv.org/abs/2201.10943 · HTML: https://mlsanity.com/paper/2201.10943/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 67 references that are in the map.
- First posted on arXiv 25 Jan 2022.

## Builds on (most cited first, 30 of 35)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks](https://mlsanity.com/paper/1706.02609/) (2017)
- [High Speed and High Dynamic Range Video with an Event Camera](https://mlsanity.com/paper/1906.07165/) (2019)
- [Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks](https://mlsanity.com/paper/2007.05785/) (2020)
- [Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion](https://mlsanity.com/paper/1812.08156/) (2018)
- [Going Deeper With Directly-Trained Larger Spiking Neural Networks](https://mlsanity.com/paper/2011.05280/) (2020)
- [SLAYER: Spike Layer Error Reassignment in Time](https://mlsanity.com/paper/1810.08646/) (2018)
- [The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception](https://mlsanity.com/paper/1801.10202/) (2018)
- [Learning Blind Video Temporal Consistency](https://mlsanity.com/paper/1808.00449/) (2018)
- [Training Deep Spiking Neural Networks using Backpropagation](https://mlsanity.com/paper/1608.08782/) (2016)
- [The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM](https://mlsanity.com/paper/1610.08336/) (2016)
- [Events-to-Video: Bringing Modern Computer Vision to Event Cameras](https://mlsanity.com/paper/1904.08298/) (2019)
- [VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows](https://mlsanity.com/paper/2108.05015/) (2021)
- [Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy](https://mlsanity.com/paper/2009.08283/) (2020)
- [Enabling Spike-based Backpropagation for Training Deep Neural Network Architectures](https://mlsanity.com/paper/1903.06379/) (2019)
- [Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks](https://mlsanity.com/paper/2106.01862/) (2021)
- [Event-based High Dynamic Range Image and Very High Frame Rate Video Generation using Conditional Generative Adversarial Networks](https://mlsanity.com/paper/1811.08230/) (2018)
- [Continuous-time Intensity Estimation Using Event Cameras](https://mlsanity.com/paper/1811.00386/) (2018)
- [LIAF-Net: Leaky Integrate and Analog Fire Network for Lightweight and Efficient Spatiotemporal Information Processing](https://mlsanity.com/paper/2011.06176/) (2020)
- [HFirst: A Temporal Approach to Object Recognition](https://mlsanity.com/paper/1508.01176/) (2015)
- [Real-Time Intensity-Image Reconstruction for Event Cameras Using Manifold Regularisation](https://mlsanity.com/paper/1607.06283/) (2016)
- [Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation: From Events to Global Motion Perception](https://mlsanity.com/paper/1807.10936/) (2018)
- [Event-Based Angular Velocity Regression with Spiking Networks](https://mlsanity.com/paper/2003.02790/) (2020)
- [A Retina-inspired Sampling Method for Visual Texture Reconstruction](https://mlsanity.com/paper/1907.08769/) (2019)
- [Hybrid Macro/Micro Level Backpropagation for Training Deep Spiking Neural Networks](https://mlsanity.com/paper/1805.07866/) (2018)

## Built on by (most cited first, 30 of 59)

- [Deep Directly-Trained Spiking Neural Networks for Object Detection](https://mlsanity.com/paper/2307.11411/) (2023)
- [Recent Event Camera Innovations: A Survey](https://mlsanity.com/paper/2408.13627/) (2024)
- [HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors](https://mlsanity.com/paper/2211.09648/) (2022)
- [OpenESS: Event-based Semantic Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2405.05259/) (2024)
- [HyperE2VID: Improving Event-Based Video Reconstruction via Hypernetworks](https://mlsanity.com/paper/2305.06382/) (2023)
- [Seeing Motion at Nighttime with an Event Camera](https://mlsanity.com/paper/2404.11884/) (2024)
- [Direct Learning-Based Deep Spiking Neural Networks: A Review](https://mlsanity.com/paper/2305.19725/) (2023)
- [V2CE: Video to Continuous Events Simulator](https://mlsanity.com/paper/2309.08891/) (2023)
- [EVREAL: Towards a Comprehensive Benchmark and Analysis Suite for Event-based Video Reconstruction](https://mlsanity.com/paper/2305.00434/) (2023)
- [SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognition](https://mlsanity.com/paper/2308.04369/) (2023)
- [SFOD: Spiking Fusion Object Detector](https://mlsanity.com/paper/2403.15192/) (2024)
- [EvGGS: A Collaborative Learning Framework for Event-based Generalizable Gaussian Splatting](https://mlsanity.com/paper/2405.14959/) (2024)
- [EventAid: Benchmarking Event-aided Image/Video Enhancement Algorithms with Real-captured Hybrid Dataset](https://mlsanity.com/paper/2312.08220/) (2023)
- [Learning Spatial-Frequency Transformer for Visual Object Tracking](https://mlsanity.com/paper/2208.08829/) (2022)
- [DailyDVS-200: A Comprehensive Benchmark Dataset for Event-Based Action Recognition](https://mlsanity.com/paper/2407.05106/) (2024)
- [Person Re-Identification without Identification via Event Anonymization](https://mlsanity.com/paper/2308.04402/) (2023)
- [EventHDR: from Event to High-Speed HDR Videos and Beyond](https://mlsanity.com/paper/2409.17029/) (2024)
- [A Hybrid ANN-SNN Architecture for Low-Power and Low-Latency Visual Perception](https://mlsanity.com/paper/2303.14176/) (2023)
- [LED: A Large-scale Real-world Paired Dataset for Event Camera Denoising](https://mlsanity.com/paper/2405.19718/) (2024)
- [Temporal Residual Guided Diffusion Framework for Event-Driven Video Reconstruction](https://mlsanity.com/paper/2407.10636/) (2024)
- [Accurate and Efficient Event-based Semantic Segmentation Using Adaptive Spiking Encoder-Decoder Network](https://mlsanity.com/paper/2304.11857/) (2023)
- [Towards Robust Event-based Networks for Nighttime via Unpaired Day-to-Night Event Translation](https://mlsanity.com/paper/2407.10703/) (2024)
- [SiGNN: A Spike-induced Graph Neural Network for Dynamic Graph Representation Learning](https://mlsanity.com/paper/2404.07941/) (2024)
- [Spiking Neural Network as Adaptive Event Stream Slicer](https://mlsanity.com/paper/2410.02249/) (2024)
- [Adaptive Event Stream Slicing for Open-Vocabulary Event-Based Object Detection via Vision-Language Knowledge Distillation](https://mlsanity.com/paper/2510.00681/) (2025)
- [A Dynamic Graph CNN with Cross-Representation Distillation for Event-Based Recognition](https://mlsanity.com/paper/2302.04177/) (2023)
- [Learning Bottleneck Transformer for Event Image-Voxel Feature Fusion based Classification](https://mlsanity.com/paper/2308.11937/) (2023)
- [SpikingNeRF: Making Bio-inspired Neural Networks See through the Real World](https://mlsanity.com/paper/2309.10987/) (2023)
- [Fully Spiking Neural Networks for Unified Frame-Event Object Tracking](https://mlsanity.com/paper/2505.20834/) (2025)
- [Deep Multi-Threshold Spiking-UNet for Image Processing](https://mlsanity.com/paper/2307.10974/) (2023)

## Abstract (arXiv)

Neuromorphic vision sensor is a new bio-inspired imaging paradigm that reports asynchronous, continuously per-pixel brightness changes called `events' with high temporal resolution and high dynamic range. So far, the event-based image reconstruction methods are based on artificial neural networks (ANN) or hand-crafted spatiotemporal smoothing techniques. In this paper, we first implement the image reconstruction work via fully spiking neural network (SNN) architecture. As the bio-inspired neural networks, SNNs operating with asynchronous binary spikes distributed over time, can potentially lead to greater computational efficiency on event-driven hardware. We propose a novel Event-based Video reconstruction framework based on a fully Spiking Neural Network (EVSNN), which utilizes Leaky-Integrate-and-Fire (LIF) neuron and Membrane Potential (MP) neuron. We find that the spiking neurons have the potential to store useful temporal information (memory) to complete such time-dependent tasks. Furthermore, to better utilize the temporal information, we propose a hybrid potential-assisted framework (PA-EVSNN) using the membrane potential of spiking neuron. The proposed neuron is referred as Adaptive Membrane Potential (AMP) neuron, which adaptively updates the membrane potential according to the input spikes. The experimental results demonstrate that our models achieve comparable performance to ANN-based models on IJRR, MVSEC, and HQF datasets. The energy consumptions of EVSNN and PA-EVSNN are 19.36× and 7.75× more computationally efficient than their ANN architectures, respectively.