# ALERT-Transformer: Bridging Asynchronous and Synchronous Machine Learning for Real-Time Event-based Spatio-Temporal Data

arXiv: https://arxiv.org/abs/2402.01393 · HTML: https://mlsanity.com/paper/2402.01393/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 2 Feb 2024.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling](https://mlsanity.com/paper/2111.14819/) (2021)
- [EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras](https://mlsanity.com/paper/1802.06898/) (2018)
- [HATS: Histograms of Averaged Time Surfaces for Robust Event-based Object Classification](https://mlsanity.com/paper/1803.07913/) (2018)
- [End-to-End Learning of Representations for Asynchronous Event-Based Data](https://mlsanity.com/paper/1904.08245/) (2019)
- [Event-based Vision meets Deep Learning on Steering Prediction for Self-driving Cars](https://mlsanity.com/paper/1804.01310/) (2018)
- [Event-based Asynchronous Sparse Convolutional Networks](https://mlsanity.com/paper/2003.09148/) (2020)
- [GET: Group Event Transformer for Event-Based Vision](https://mlsanity.com/paper/2310.02642/) (2023)
- [EventNet: Asynchronous Recursive Event Processing](https://mlsanity.com/paper/1812.07045/) (2018)
- [Asynchronous Convolutional Networks for Object Detection in Neuromorphic Cameras](https://mlsanity.com/paper/1805.07931/) (2018)
- [Temporal Binary Representation for Event-Based Action Recognition](https://mlsanity.com/paper/2010.08946/) (2020)
- [Event Transformer. A sparse-aware solution for efficient event data processing](https://mlsanity.com/paper/2204.03355/) (2022)
- [Efficient Human Pose Estimation via 3D Event Point Cloud](https://mlsanity.com/paper/2206.04511/) (2022)
- [Exploiting Spatial Sparsity for Event Cameras with Visual Transformers](https://mlsanity.com/paper/2202.05054/) (2022)
- [TinyTracker: Ultra-Fast and Ultra-Low-Power Edge Vision In-Sensor for Gaze Estimation](https://mlsanity.com/paper/2307.07813/) (2023)
- [Ultra-Efficient On-Device Object Detection on AI-Integrated Smart Glasses with TinyissimoYOLO](https://mlsanity.com/paper/2311.01057/) (2023)
- [Event Transformer+. A multi-purpose solution for efficient event data processing](https://mlsanity.com/paper/2211.12222/) (2022)
- [Low-power event-based face detection with asynchronous neuromorphic hardware](https://mlsanity.com/paper/2312.14261/) (2023)

## Built on by (most cited first, 8 of 8)

- [Scalable Event-by-event Processing of Neuromorphic Sensory Signals With Deep State-Space Models](https://mlsanity.com/paper/2404.18508/) (2024)
- [Compute-in-Memory Implementation of State Space Models for Event Sequence Processing](https://mlsanity.com/paper/2511.13912/) (2025)
- [Low-Latency Event-Based Object Detection with Spatially-Sparse Linear Attention](https://mlsanity.com/paper/2603.06228/) (2026)
- [Learning to Match Unpaired Data with Minimum Entropy Coupling](https://mlsanity.com/paper/2503.08501/) (2025)
- [Maximizing Asynchronicity in Event-based Neural Networks](https://mlsanity.com/paper/2505.11165/) (2025)
- [Mitigating Data Scarcity in Time Series Analysis: A Foundation Model with Series-Symbol Data Generation](https://mlsanity.com/paper/2502.15466/) (2025)
- [FLAMES: A Hybrid Spiking-State Space Model for Adaptive Memory Retention in Event-Based Learning](https://mlsanity.com/paper/2504.01257/) (2025)
- [FATE: Pillar Encoding and Frequency-Aware Training for Event-Based Object Detection](https://mlsanity.com/paper/2606.17334/) (2026)

## Abstract (arXiv)

We seek to enable classic processing of continuous ultra-sparse spatiotemporal data generated by event-based sensors with dense machine learning models. We propose a novel hybrid pipeline composed of asynchronous sensing and synchronous processing that combines several ideas: (1) an embedding based on PointNet models -- the ALERT module -- that can continuously integrate new and dismiss old events thanks to a leakage mechanism, (2) a flexible readout of the embedded data that allows to feed any downstream model with always up-to-date features at any sampling rate, (3) exploiting the input sparsity in a patch-based approach inspired by Vision Transformer to optimize the efficiency of the method. These embeddings are then processed by a transformer model trained for object and gesture recognition. Using this approach, we achieve performances at the state-of-the-art with a lower latency than competitors. We also demonstrate that our asynchronous model can operate at any desired sampling rate.