# Scene Adaptive Sparse Transformer for Event-based Object Detection

arXiv: https://arxiv.org/abs/2404.01882 · HTML: https://mlsanity.com/paper/2404.01882/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 68 references that are in the map.
- First posted on arXiv 2 Apr 2024.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [MaxViT: Multi-Axis Vision Transformer](https://mlsanity.com/paper/2204.01697/) (2022)
- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks](https://mlsanity.com/paper/2007.05785/) (2020)
- [Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion](https://mlsanity.com/paper/1812.08156/) (2018)
- [LocalViT: Analyzing Locality in Vision Transformers](https://mlsanity.com/paper/2104.05707/) (2021)
- [HATS: Histograms of Averaged Time Surfaces for Robust Event-based Object Classification](https://mlsanity.com/paper/1803.07913/) (2018)
- [End-to-End Learning of Representations for Asynchronous Event-Based Data](https://mlsanity.com/paper/1904.08245/) (2019)
- [cosFormer: Rethinking Softmax in Attention](https://mlsanity.com/paper/2202.08791/) (2022)
- [Training Deep Spiking Neural Networks using Backpropagation](https://mlsanity.com/paper/1608.08782/) (2016)
- [Learning A Sparse Transformer Network for Effective Image Deraining](https://mlsanity.com/paper/2303.11950/) (2023)
- [Learning to Detect Objects with a 1 Megapixel Event Camera](https://mlsanity.com/paper/2009.13436/) (2020)
- [Chasing Sparsity in Vision Transformers: An End-to-End Exploration](https://mlsanity.com/paper/2106.04533/) (2021)
- [SPViT: Enabling Faster Vision Transformers via Soft Token Pruning](https://mlsanity.com/paper/2112.13890/) (2021)
- [Recurrent Vision Transformers for Object Detection with Event Cameras](https://mlsanity.com/paper/2212.05598/) (2022)

## Built on by (most cited first, 30 of 39)

- [Recent Event Camera Innovations: A Survey](https://mlsanity.com/paper/2408.13627/) (2024)
- [Object Detection using Event Camera: A MoE Heat Conduction based Detector and A New Benchmark Dataset](https://mlsanity.com/paper/2412.06647/) (2024)
- [Event-based Tiny Object Detection: A Benchmark Dataset and Baseline](https://mlsanity.com/paper/2506.23575/) (2025)
- [FlexEvent: Towards Flexible Event-Frame Object Detection at Varying Operational Frequencies](https://mlsanity.com/paper/2412.06708/) (2024)
- [EventFly: Event Camera Perception from Ground to the Sky](https://mlsanity.com/paper/2503.19916/) (2025)
- [A Recurrent YOLOv8-based framework for Event-Based Object Detection](https://mlsanity.com/paper/2408.05321/) (2024)
- [EvRT-DETR: Latent Space Adaptation of Image Detectors for Event-based Vision](https://mlsanity.com/paper/2412.02890/) (2024)
- [Helios: An extremely low power event-based gesture recognition for always-on smart eyewear](https://mlsanity.com/paper/2407.05206/) (2024)
- [Ev-3DOD: Pushing the Temporal Boundaries of 3D Object Detection with Event Cameras](https://mlsanity.com/paper/2502.19630/) (2025)
- [Frequency-Adaptive Low-Latency Object Detection Using Events and Frames](https://mlsanity.com/paper/2412.04149/) (2024)
- [MutualForce: Mutual-Aware Enhancement for 4D Radar-LiDAR 3D Object Detection](https://mlsanity.com/paper/2501.10266/) (2025)
- [Event Camera Meets Mobile Embodied Perception: Abstraction, Algorithm, Acceleration, Application](https://mlsanity.com/paper/2503.22943/) (2025)
- [Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras](https://mlsanity.com/paper/2507.17664/) (2025)
- [Beyond conventional vision: RGB-event fusion for robust object detection in dynamic traffic scenarios](https://mlsanity.com/paper/2508.10704/) (2025)
- [A Survey on Dynamic Neural Networks: from Computer Vision to Multi-modal Sensor Fusion](https://mlsanity.com/paper/2501.07451/) (2025)
- [When Person Re-Identification Meets Event Camera: A Benchmark Dataset and An Attribute-guided Re-Identification Framework](https://mlsanity.com/paper/2507.13659/) (2025)
- [Unleashing the Temporal Potential of Stereo Event Cameras for Continuous-Time 3D Object Detection](https://mlsanity.com/paper/2508.02288/) (2025)
- [ESTR-CoT: Towards Explainable and Accurate Event Stream based Scene Text Recognition with Chain-of-Thought Reasoning](https://mlsanity.com/paper/2507.02200/) (2025)
- [EventFlash: Towards Efficient MLLMs for Event-Based Vision](https://mlsanity.com/paper/2602.03230/) (2026)
- [Low-Latency Event-Based Object Detection with Spatially-Sparse Linear Attention](https://mlsanity.com/paper/2603.06228/) (2026)
- [Event-based Graph Representation with Spatial and Motion Vectors for Asynchronous Object Detection](https://mlsanity.com/paper/2507.15150/) (2025)
- [Spiking Patches: Asynchronous, Sparse, and Efficient Tokens for Event Cameras](https://mlsanity.com/paper/2510.26614/) (2025)
- [EventDrive: Event Cameras for Vision-Language Driving Intelligence](https://mlsanity.com/paper/2606.18242/) (2026)
- [Helios 2.0: A Robust, Ultra-Low Power Gesture Recognition System Optimised for Event-Sensor based Wearables](https://mlsanity.com/paper/2503.07825/) (2025)
- [Dynamic Graph Induced Contour-aware Heat Conduction Network for Event-based Object Detection](https://mlsanity.com/paper/2505.12908/) (2025)
- [Focus Through Motion: RGB-Event Collaborative Token Sparsification for Efficient Object Detection](https://mlsanity.com/paper/2509.03872/) (2025)
- [Rethinking Event-Based Object Detection through Representation-Level Temporal Aggregation and Model-Level Hypergraph Reasoning](https://mlsanity.com/paper/2605.08825/) (2026)
- [M2E-UAV: A Benchmark and Analysis for Onboard Motion-on-Motion Event-Based Tiny UAV Detection](https://mlsanity.com/paper/2605.10496/) (2026)
- [Enhancing Event-based Object Detection with Monocular Normal Maps](https://mlsanity.com/paper/2508.02127/) (2025)
- [CETUS: Causal Event-Driven Temporal Modeling With Unified Variable-Rate Scheduling](https://mlsanity.com/paper/2509.13784/) (2025)

## Abstract (arXiv)

While recent Transformer-based approaches have shown impressive performances on event-based object detection tasks, their high computational costs still diminish the low power consumption advantage of event cameras. Image-based works attempt to reduce these costs by introducing sparse Transformers. However, they display inadequate sparsity and adaptability when applied to event-based object detection, since these approaches cannot balance the fine granularity of token-level sparsification and the efficiency of window-based Transformers, leading to reduced performance and efficiency. Furthermore, they lack scene-specific sparsity optimization, resulting in information loss and a lower recall rate. To overcome these limitations, we propose the Scene Adaptive Sparse Transformer (SAST). SAST enables window-token co-sparsification, significantly enhancing fault tolerance and reducing computational overhead. Leveraging the innovative scoring and selection modules, along with the Masked Sparse Window Self-Attention, SAST showcases remarkable scene-aware adaptability: It focuses only on important objects and dynamically optimizes sparsity level according to scene complexity, maintaining a remarkable balance between performance and computational cost. The evaluation results show that SAST outperforms all other dense and sparse networks in both performance and efficiency on two large-scale event-based object detection datasets (1Mpx and Gen1). Code: https://github.com/Peterande/SAST