# State Space Models for Event Cameras

arXiv: https://arxiv.org/abs/2402.15584 · HTML: https://mlsanity.com/paper/2402.15584/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 49 references that are in the map.
- First posted on arXiv 23 Feb 2024.

## Builds on (most cited first, 30 of 40)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [On the Parameterization and Initialization of Diagonal State Space Models](https://mlsanity.com/paper/2206.11893/) (2022)
- [Diagonal State Spaces are as Effective as Structured State Spaces](https://mlsanity.com/paper/2203.14343/) (2022)
- [DSEC: A Stereo Event Camera Dataset for Driving Scenarios](https://mlsanity.com/paper/2103.06011/) (2021)
- [YOLOv6: A Single-Stage Object Detection Framework for Industrial Applications](https://mlsanity.com/paper/2209.02976/) (2022)
- [Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates](https://mlsanity.com/paper/1708.07120/) (2017)
- [Learning to Detect Objects with a 1 Megapixel Event Camera](https://mlsanity.com/paper/2009.13436/) (2020)
- [It's Raw! Audio Generation with State-Space Models](https://mlsanity.com/paper/2202.09729/) (2022)
- [Recurrent Vision Transformers for Object Detection with Event Cameras](https://mlsanity.com/paper/2212.05598/) (2022)
- [TimeLens: Event-based Video Frame Interpolation](https://mlsanity.com/paper/2106.07286/) (2021)
- [CKConv: Continuous Kernel Convolution For Sequential Data](https://mlsanity.com/paper/2102.02611/) (2021)
- [AEGNN: Asynchronous Event-based Graph Neural Networks](https://mlsanity.com/paper/2203.17149/) (2022)
- [A Large Scale Event-based Detection Dataset for Automotive](https://mlsanity.com/paper/2001.08499/) (2020)
- [Event-based Asynchronous Sparse Convolutional Networks](https://mlsanity.com/paper/2003.09148/) (2020)
- [Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding](https://mlsanity.com/paper/2105.12723/) (2021)
- [Object Detection with Spiking Neural Networks on Automotive Event Data](https://mlsanity.com/paper/2205.04339/) (2022)
- [GET: Group Event Transformer for Event-Based Vision](https://mlsanity.com/paper/2310.02642/) (2023)
- [FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes](https://mlsanity.com/paper/2110.08059/) (2021)
- [Asynchronous Convolutional Networks for Object Detection in Neuromorphic Cameras](https://mlsanity.com/paper/1805.07931/) (2018)

## Built on by (most cited first, 30 of 75)

- [Recent Event Camera Innovations: A Survey](https://mlsanity.com/paper/2408.13627/) (2024)
- [State Space Model for New-Generation Network Alternative to Transformers: A Survey](https://mlsanity.com/paper/2404.09516/) (2024)
- [A Survey on Visual Mamba](https://mlsanity.com/paper/2404.15956/) (2024)
- [Scalable Event-by-event Processing of Neuromorphic Sensory Signals With Deep State-Space Models](https://mlsanity.com/paper/2404.18508/) (2024)
- [Embracing Events and Frames with Hierarchical Feature Refinement Network for Object Detection](https://mlsanity.com/paper/2407.12582/) (2024)
- [Object Detection using Event Camera: A MoE Heat Conduction based Detector and A New Benchmark Dataset](https://mlsanity.com/paper/2412.06647/) (2024)
- [Mamba-FETrack: Frame-Event Tracking via State Space Model](https://mlsanity.com/paper/2404.18174/) (2024)
- [Rethinking Efficient and Effective Point-based Networks for Event Camera Classification and Regression: EventMamba](https://mlsanity.com/paper/2405.06116/) (2024)
- [Limits of Deep Learning: Sequence Modeling through the Lens of Complexity Theory](https://mlsanity.com/paper/2405.16674/) (2024)
- [S7: Selective and Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2410.03464/) (2024)
- [LLEMamba: Low-Light Enhancement via Relighting-Guided Mamba with Deep Unfolding Network](https://mlsanity.com/paper/2406.01028/) (2024)
- [Integrating Extra Modality Helps Segmentor Find Camouflaged Objects Well](https://mlsanity.com/paper/2502.14471/) (2025)
- [FlexEvent: Towards Flexible Event-Frame Object Detection at Varying Operational Frequencies](https://mlsanity.com/paper/2412.06708/) (2024)
- [A Recurrent YOLOv8-based framework for Event-Based Object Detection](https://mlsanity.com/paper/2408.05321/) (2024)
- [EvRT-DETR: Latent Space Adaptation of Image Detectors for Event-based Vision](https://mlsanity.com/paper/2412.02890/) (2024)
- [MambaPupil: Bidirectional Selective Recurrent model for Event-based Eye tracking](https://mlsanity.com/paper/2404.12083/) (2024)
- [Frequency-Adaptive Low-Latency Object Detection Using Events and Frames](https://mlsanity.com/paper/2412.04149/) (2024)
- [Efficient Event-Based Object Detection: A Hybrid Neural Network with Spatial and Temporal Attention](https://mlsanity.com/paper/2403.10173/) (2024)
- [AC-MAMBASEG: An adaptive convolution and Mamba-based architecture for enhanced skin lesion segmentation](https://mlsanity.com/paper/2405.03011/) (2024)
- [Zero-Shot Temporal Resolution Domain Adaptation for Spiking Neural Networks](https://mlsanity.com/paper/2411.04760/) (2024)
- [EventMamba: Enhancing Spatio-Temporal Locality with State Space Models for Event-Based Video Reconstruction](https://mlsanity.com/paper/2503.19721/) (2025)
- [Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras](https://mlsanity.com/paper/2507.17664/) (2025)
- [PRE-Mamba: A 4D State Space Model for Ultra-High-Frequent Event Camera Deraining](https://mlsanity.com/paper/2505.05307/) (2025)
- [Deep Event-based Object Detection in Autonomous Driving: A Survey](https://mlsanity.com/paper/2405.03995/) (2024)
- [PLEIADES: Building Temporal Kernels with Orthogonal Polynomials](https://mlsanity.com/paper/2405.12179/) (2024)
- [Sign Language Translation using Frame and Event Stream: Benchmark Dataset and Algorithms](https://mlsanity.com/paper/2503.06484/) (2025)
- [State-Space Model Inspired Multiple-Input Multiple-Output Spiking Neurons](https://mlsanity.com/paper/2504.02591/) (2025)
- [TESPEC: Temporally-Enhanced Self-Supervised Pretraining for Event Cameras](https://mlsanity.com/paper/2508.00913/) (2025)
- [Generative Event Pretraining with Foundation Model Alignment](https://mlsanity.com/paper/2603.23032/) (2026)
- [GG-SSMs: Graph-Generating State Space Models](https://mlsanity.com/paper/2412.12423/) (2024)

## Abstract (arXiv)

Today, state-of-the-art deep neural networks that process event-camera data first convert a temporal window of events into dense, grid-like input representations. As such, they exhibit poor generalizability when deployed at higher inference frequencies (i.e., smaller temporal windows) than the ones they were trained on. We address this challenge by introducing state-space models (SSMs) with learnable timescale parameters to event-based vision. This design adapts to varying frequencies without the need to retrain the network at different frequencies. Additionally, we investigate two strategies to counteract aliasing effects when deploying the model at higher frequencies. We comprehensively evaluate our approach against existing methods based on RNN and Transformer architectures across various benchmarks, including Gen1 and 1 Mpx event camera datasets. Our results demonstrate that SSM-based models train 33% faster and also exhibit minimal performance degradation when tested at higher frequencies than the training input. Traditional RNN and Transformer models exhibit performance drops of more than 20 mAP, with SSMs having a drop of 3.76 mAP, highlighting the effectiveness of SSMs in event-based vision tasks.