# Event-based Action Recognition by Pre-Training on Multi-Frame Representations

arXiv: https://arxiv.org/abs/2610.13856 · HTML: https://mlsanity.com/paper/2610.13856/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 42 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Builds on (most cited first, 30 of 31)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [Video Transformer Network](https://mlsanity.com/paper/2102.00719/) (2021)
- [Human Action Recognition and Prediction: A Survey](https://mlsanity.com/paper/1806.11230/) (2018)
- [Event-based Vision meets Deep Learning on Steering Prediction for Self-driving Cars](https://mlsanity.com/paper/1804.01310/) (2018)
- [Event-based Moving Object Detection and Tracking](https://mlsanity.com/paper/1803.04523/) (2018)
- [Temporal Binary Representation for Event-Based Action Recognition](https://mlsanity.com/paper/2010.08946/) (2020)
- [Event Transformer. A sparse-aware solution for efficient event data processing](https://mlsanity.com/paper/2204.03355/) (2022)
- [Event Camera Data Pre-training](https://mlsanity.com/paper/2301.01928/) (2023)
- [HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors](https://mlsanity.com/paper/2211.09648/) (2022)
- [AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoders](https://mlsanity.com/paper/2211.09120/) (2022)
- [Masked Event Modeling: Self-Supervised Pretraining for Event Cameras](https://mlsanity.com/paper/2212.10368/) (2022)
- [EventBind: Learning a Unified Representation to Bind Them All for Event-based Open-world Understanding](https://mlsanity.com/paper/2308.03135/) (2023)
- [Rethinking Efficient and Effective Point-based Networks for Event Camera Classification and Regression: EventMamba](https://mlsanity.com/paper/2405.06116/) (2024)
- [DailyDVS-200: A Comprehensive Benchmark Dataset for Event-Based Action Recognition](https://mlsanity.com/paper/2407.05106/) (2024)
- [SMAUG: Sparse Masked Autoencoder for Efficient Video-Language Pre-training](https://mlsanity.com/paper/2211.11446/) (2022)
- [Event Stream based Human Action Recognition: A High-Definition Benchmark Dataset and Algorithms](https://mlsanity.com/paper/2408.09764/) (2024)
- [EventTransAct: A video transformer-based framework for Event-camera based action recognition](https://mlsanity.com/paper/2308.13711/) (2023)
- [Data-efficient Event Camera Pre-training via Disentangled Masked Modeling](https://mlsanity.com/paper/2403.00416/) (2024)
- [Event Masked Autoencoder: Point-wise Action Recognition with Event-Based Cameras](https://mlsanity.com/paper/2501.01040/) (2025)
- [Generative Event Pretraining with Foundation Model Alignment](https://mlsanity.com/paper/2603.23032/) (2026)
- [Revealing Latent Information: A Physics-inspired Self-supervised Pre-training Framework for Noisy and Sparse Events](https://mlsanity.com/paper/2508.05507/) (2025)
- [DarkShake-DVS: Event-based Human Action Recognition under Low-light andShaking Camera Conditions](https://mlsanity.com/paper/2605.20680/) (2026)
- [PASS: Path-selective State Space Model for Event-based Recognition](https://mlsanity.com/paper/2409.16953/) (2024)
- [SMV-EAR: Bring Spatiotemporal Multi-View Representation Learning into Efficient Event-Based Action Recognition](https://mlsanity.com/paper/2601.17391/) (2026)

## Abstract (arXiv)

Action recognition is a crucial task in domains like robotics, patient monitoring, or surveillance. Recently, event-based action recognition has attracted much attention due to the event camera's ability to capture fast motions and its improved privacy over conventional vision cameras. While frame-based event representations work well for short event sequences, their discriminative ability declines sharply with longer and more complex event sequences. In this work, we introduce a multi-frame representation that greatly improves recognition performance over its single-frame counterpart. We adopt a masked auto-encoding framework to pre-train a Vision Transformer using a sparsity-aware masking algorithm and token selection mechanism to improve downstream performance and control computational complexity. We also introduce multiple mask tokens for the decoder with a position-aware attention mechanism, leading to further improved performance. Our proposed method shows impressive performance for challenging long-duration and complex event sequences, yielding state-of-the-art performance on THU-EACT-50 CHL and DailyDVS-200.