# Representation Learning for Event-based Visuomotor Policies

arXiv: https://arxiv.org/abs/2103.00806 · HTML: https://mlsanity.com/paper/2103.00806/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 50 references that are in the map.
- First posted on arXiv 1 Mar 2021.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images](https://mlsanity.com/paper/1506.07365/) (2015)
- [EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras](https://mlsanity.com/paper/1802.06898/) (2018)
- [HATS: Histograms of Averaged Time Surfaces for Robust Event-based Object Classification](https://mlsanity.com/paper/1803.07913/) (2018)
- [End-to-End Learning of Representations for Asynchronous Event-Based Data](https://mlsanity.com/paper/1904.08245/) (2019)
- [Event-based Vision meets Deep Learning on Steering Prediction for Self-driving Cars](https://mlsanity.com/paper/1804.01310/) (2018)
- [Event-based Asynchronous Sparse Convolutional Networks](https://mlsanity.com/paper/2003.09148/) (2020)
- [Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences](https://mlsanity.com/paper/1610.09513/) (2016)
- [Does computer vision matter for action?](https://mlsanity.com/paper/1905.12887/) (2019)
- [EventNet: Asynchronous Recursive Event Processing](https://mlsanity.com/paper/1812.07045/) (2018)
- [Asynchronous Convolutional Networks for Object Detection in Neuromorphic Cameras](https://mlsanity.com/paper/1805.07931/) (2018)
- [Neuromorphic Deep Learning Machines](https://mlsanity.com/paper/1612.05596/) (2016)
- [Event Probability Mask (EPM) and Event Denoising Convolutional Neural Network (EDnCNN) for Neuromorphic Cameras](https://mlsanity.com/paper/2003.08282/) (2020)
- [Reinforcement co-Learning of Deep and Spiking Neural Networks for Energy-Efficient Mapless Navigation with Neuromorphic Hardware](https://mlsanity.com/paper/2003.01157/) (2020)
- [Towards Low-Latency High-Bandwidth Control of Quadrotors using Event Cameras](https://mlsanity.com/paper/1911.04553/) (2019)
- [Vertical Landing for Micro Air Vehicles using Event-Based Optical Flow](https://mlsanity.com/paper/1702.00061/) (2017)
- [Playing Atari with Six Neurons](https://mlsanity.com/paper/1806.01363/) (2018)
- [Evolved Neuromorphic Control for High Speed Divergence-based Landings of MAVs](https://mlsanity.com/paper/2003.03118/) (2020)
- [Exploration of Reinforcement Learning for Event Camera using Car-like Robots](https://mlsanity.com/paper/2004.00801/) (2020)

## Built on by (most cited first, 20 of 20)

- [Event Transformer. A sparse-aware solution for efficient event data processing](https://mlsanity.com/paper/2204.03355/) (2022)
- [Event Transformer+. A multi-purpose solution for efficient event data processing](https://mlsanity.com/paper/2211.12222/) (2022)
- [Event Stream based Human Action Recognition: A High-Definition Benchmark Dataset and Algorithms](https://mlsanity.com/paper/2408.09764/) (2024)
- [Unsupervised Optical Flow Estimation with Dynamic Timing Representation for Spike Camera](https://mlsanity.com/paper/2307.06003/) (2023)
- [SEBVS: Synthetic Event-based Visual Servoing for Robot Navigation and Manipulation](https://mlsanity.com/paper/2508.17643/) (2025)
- [Point-Voxel Absorbing Graph Representation Learning for Event Stream based Recognition](https://mlsanity.com/paper/2306.05239/) (2023)
- [Retain, Blend, and Exchange: A Quality-aware Spatial-Stereo Fusion Approach for Event Stream Recognition](https://mlsanity.com/paper/2406.18845/) (2024)
- [Hardware, Algorithms, and Applications of the Neuromorphic Vision Sensor: a Review](https://mlsanity.com/paper/2504.08588/) (2025)
- [EventFlash: Towards Efficient MLLMs for Event-Based Vision](https://mlsanity.com/paper/2602.03230/) (2026)
- [Text-to-Events: Synthetic Event Camera Streams from Conditional Text Input](https://mlsanity.com/paper/2406.03439/) (2024)
- [Learning to Remove Lens Flare in Event Camera](https://mlsanity.com/paper/2512.09016/) (2025)
- [CERiL: Continuous Event-based Reinforcement Learning](https://mlsanity.com/paper/2302.07667/) (2023)
- [Uncertainty-aware Bridge based Mobile-Former Network for Event-based Pattern Recognition](https://mlsanity.com/paper/2401.11123/) (2024)
- [Leveraging Event Streams with Deep Reinforcement Learning for End-to-End UAV Tracking](https://mlsanity.com/paper/2410.14685/) (2024)
- [Event Fusion Photometric Stereo Network](https://mlsanity.com/paper/2303.00308/) (2023)
- [Semore: VLM-guided Enhanced Semantic Motion Representations for Visual Reinforcement Learning](https://mlsanity.com/paper/2512.05172/) (2025)
- [CADENCE: Context-Adaptive Depth Estimation for Navigation and Computational Efficiency](https://mlsanity.com/paper/2604.07286/) (2026)
- [Self-Predictive Representation for Autonomous UAV Object-Goal Navigation](https://mlsanity.com/paper/2604.21130/) (2026)
- [FLEET: Token-Based Feature Extraction for Event Camera-based Reinforcement Learning](https://mlsanity.com/paper/2608.16523/) (2026)
- [ECHO: Event-Augmented Context with Hindsight and Outlook for Wrist-Only Manipulation](https://mlsanity.com/paper/2609.34893/) (2026)

## Abstract (arXiv)

Event-based cameras are dynamic vision sensors that provide asynchronous measurements of changes in per-pixel brightness at a microsecond level. This makes them significantly faster than conventional frame-based cameras, and an appealing choice for high-speed navigation. While an interesting sensor modality, this asynchronously streamed event data poses a challenge for machine learning techniques that are more suited for frame-based data. In this paper, we present an event variational autoencoder and show that it is feasible to learn compact representations directly from asynchronous spatiotemporal event data. Furthermore, we show that such pretrained representations can be used for event-based reinforcement learning instead of end-to-end reward driven perception. We validate this framework of learning event-based visuomotor policies by applying it to an obstacle avoidance scenario in simulation. Compared to techniques that treat event data as images, we show that representations learnt from event streams result in faster policy training, adapt to different control capacities, and demonstrate a higher degree of robustness.