# Exploiting Spatial Sparsity for Event Cameras with Visual Transformers

arXiv: https://arxiv.org/abs/2202.05054 · HTML: https://mlsanity.com/paper/2202.05054/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 17 references that are in the map.
- First posted on arXiv 10 Feb 2022.

## Builds on (most cited first, 14 of 14)

- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021)
- [Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades](https://mlsanity.com/paper/1507.07629/) (2015)
- [Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion](https://mlsanity.com/paper/1812.08156/) (2018)
- [EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras](https://mlsanity.com/paper/1802.06898/) (2018)
- [HATS: Histograms of Averaged Time Surfaces for Robust Event-based Object Classification](https://mlsanity.com/paper/1803.07913/) (2018)
- [End-to-End Learning of Representations for Asynchronous Event-Based Data](https://mlsanity.com/paper/1904.08245/) (2019)
- [E-RAFT: Dense Optical Flow from Event Cameras](https://mlsanity.com/paper/2108.10552/) (2021)
- [Efficient Training of Visual Transformers with Small Datasets](https://mlsanity.com/paper/2106.03746/) (2021)
- [NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps](https://mlsanity.com/paper/1706.01406/) (2017)
- [EventDrop: data augmentation for event-based learning](https://mlsanity.com/paper/2106.05836/) (2021)

## Built on by (most cited first, 23 of 23)

- [Recurrent Vision Transformers for Object Detection with Event Cameras](https://mlsanity.com/paper/2212.05598/) (2022)
- [GET: Group Event Transformer for Event-Based Vision](https://mlsanity.com/paper/2310.02642/) (2023)
- [Single-Model and Any-Modality for Video Object Tracking](https://mlsanity.com/paper/2311.15851/) (2023)
- [Masked Event Modeling: Self-Supervised Pretraining for Event Cameras](https://mlsanity.com/paper/2212.10368/) (2022)
- [Scene Adaptive Sparse Transformer for Event-based Object Detection](https://mlsanity.com/paper/2404.01882/) (2024)
- [EventBind: Learning a Unified Representation to Bind Them All for Event-based Open-world Understanding](https://mlsanity.com/paper/2308.03135/) (2023)
- [3ET: Efficient Event-based Eye Tracking using a Change-Based ConvLSTM Network](https://mlsanity.com/paper/2308.11771/) (2023)
- [OmniBind: Teach to Build Unequal-Scale Modality Interaction for Omni-Bind of All](https://mlsanity.com/paper/2405.16108/) (2024)
- [Event-Based Eye Tracking. AIS 2024 Challenge Survey](https://mlsanity.com/paper/2404.11770/) (2024)
- [Event Voxel Set Transformer for Spatiotemporal Representation Learning on Event Streams](https://mlsanity.com/paper/2303.03856/) (2023)
- [ALERT-Transformer: Bridging Asynchronous and Synchronous Machine Learning for Real-Time Event-based Spatio-Temporal Data](https://mlsanity.com/paper/2402.01393/) (2024)
- [Event-Based Eye Tracking. 2025 Event-based Vision Workshop](https://mlsanity.com/paper/2504.18249/) (2025)
- [Neuromorphic Imaging and Classification with Graph Learning](https://mlsanity.com/paper/2309.15627/) (2023)
- [EventSleep: Sleep Activity Recognition with Event Cameras](https://mlsanity.com/paper/2404.01801/) (2024)
- [Motion Segmentation for Neuromorphic Aerial Surveillance](https://mlsanity.com/paper/2405.15209/) (2024)
- [Text-to-Events: Synthetic Event Camera Streams from Conditional Text Input](https://mlsanity.com/paper/2406.03439/) (2024)
- [Event USKT : U-State Space Model in Knowledge Transfer for Event Cameras](https://mlsanity.com/paper/2411.15276/) (2024)
- [Spiking Patches: Asynchronous, Sparse, and Efficient Tokens for Event Cameras](https://mlsanity.com/paper/2510.26614/) (2025)
- [Event Transformer](https://mlsanity.com/paper/2204.05172/) (2022)
- [Expanding Event Modality Applications through a Robust CLIP-Based Encoder](https://mlsanity.com/paper/2412.03093/) (2024)
- [EMF: Event Meta Formers for Event-based Real-time Traffic Object Detection](https://mlsanity.com/paper/2504.04124/) (2025)
- [Uncertainty-Weighted Fusion of Image and Synthetic Event for Video Anomaly Detection](https://mlsanity.com/paper/2505.02393/) (2025)
- [DELTA: Dense Depth from Events and LiDAR using Transformer's Attention](https://mlsanity.com/paper/2505.02593/) (2025)

## Abstract (arXiv)

Event cameras report local changes of brightness through an asynchronous stream of output events. Events are spatially sparse at pixel locations with little brightness variation. We propose using a visual transformer (ViT) architecture to leverage its ability to process a variable-length input. The input to the ViT consists of events that are accumulated into time bins and spatially separated into non-overlapping sub-regions called patches. Patches are selected when the number of nonzero pixel locations within a sub-region is above a threshold. We show that by fine-tuning a ViT model on the selected active patches, we can reduce the average number of patches fed into the backbone during the inference by at least 50% with only a minor drop (0.34%) of the classification accuracy on the N-Caltech101 dataset. This reduction translates into a decrease of 51% in Multiply-Accumulate (MAC) operations and an increase of 46% in the inference speed using a server CPU.