# MambaST: A Plug-and-Play Cross-Spectral Spatial-Temporal Fuser for Efficient Pedestrian Detection

arXiv: https://arxiv.org/abs/2408.01037 · HTML: https://mlsanity.com/paper/2408.01037/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 48 references that are in the map.
- First posted on arXiv 2 Aug 2024.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [VMamba: Visual State Space Model](https://mlsanity.com/paper/2401.10166/) (2024)
- [VideoMamba: State Space Model for Efficient Video Understanding](https://mlsanity.com/paper/2403.06977/) (2024)
- [Vision Transformer with Deformable Attention](https://mlsanity.com/paper/2201.00520/) (2022)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [Pyramid Attention Network for Semantic Segmentation](https://mlsanity.com/paper/1805.10180/) (2018)
- [TAM: Temporal Adaptive Module for Video Recognition](https://mlsanity.com/paper/2005.06803/) (2020)
- [Towards Large-Scale Small Object Detection: Survey and Benchmarks](https://mlsanity.com/paper/2207.14096/) (2022)
- [Cross-Modality Fusion Transformer for Multispectral Object Detection](https://mlsanity.com/paper/2111.00273/) (2021)
- [Multispectral Fusion for Object Detection with Cyclic Fuse-and-Refine Blocks](https://mlsanity.com/paper/2009.12664/) (2020)
- [Multispectral Deep Neural Networks for Pedestrian Detection](https://mlsanity.com/paper/1611.02644/) (2016)
- [Multimodal Object Detection via Probabilistic Ensembling](https://mlsanity.com/paper/2104.02904/) (2021)
- [Fusion-Mamba for Cross-modality Object Detection](https://mlsanity.com/paper/2404.09146/) (2024)
- [Temporally Efficient Vision Transformer for Video Instance Segmentation](https://mlsanity.com/paper/2204.08412/) (2022)
- [A Survey on Visual Mamba](https://mlsanity.com/paper/2404.15956/) (2024)
- [Multispectral Pedestrian Detection via Simultaneous Detection and Segmentation](https://mlsanity.com/paper/1808.04818/) (2018)
- [Predicting Pedestrian Crossing Intention with Feature Fusion and Spatio-Temporal Attention](https://mlsanity.com/paper/2104.05485/) (2021)
- [RGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Driving](https://mlsanity.com/paper/1906.00208/) (2019)
- [TAda! Temporally-Adaptive Convolutions for Video Understanding](https://mlsanity.com/paper/2110.06178/) (2021)
- [Spatio-Contextual Deep Network Based Multimodal Pedestrian Detection For Autonomous Driving](https://mlsanity.com/paper/2105.12713/) (2021)
- [MS-DETR: Multispectral Pedestrian Detection Transformer with Loosely Coupled Fusion and Modality-Balanced Optimization](https://mlsanity.com/paper/2302.00290/) (2023)
- [CFMW: Cross-modality Fusion Mamba for Robust Object Detection under Adverse Weather](https://mlsanity.com/paper/2404.16302/) (2024)
- [LRPD: Long Range 3D Pedestrian Detection Leveraging Specific Strengths of LiDAR and RGB](https://mlsanity.com/paper/2006.09738/) (2020)

## Built on by (most cited first, 6 of 6)

- [WaveMamba: Wavelet-Driven Mamba Fusion for RGB-Infrared Object Detection](https://mlsanity.com/paper/2507.18173/) (2025)
- [CoCMT: Communication-Efficient Cross-Modal Transformer for Collaborative Perception](https://mlsanity.com/paper/2503.13504/) (2025)
- [AirV2X: Unified Air-Ground Vehicle-to-Everything Collaboration](https://mlsanity.com/paper/2506.19283/) (2025)
- [MambaFusion: Height-Fidelity Dense Global Fusion for Multi-modal 3D Object Detection](https://mlsanity.com/paper/2507.04369/) (2025)
- [SafeCoop: Unravelling Full Stack Safety in Agentic Collaborative Driving](https://mlsanity.com/paper/2510.18123/) (2025)
- [Strip-Fusion: Spatiotemporal Fusion for Multispectral Pedestrian Detection](https://mlsanity.com/paper/2601.18008/) (2026)

## Abstract (arXiv)

This paper proposes MambaST, a plug-and-play cross-spectral spatial-temporal fusion pipeline for efficient pedestrian detection. Several challenges exist for pedestrian detection in autonomous driving applications. First, it is difficult to perform accurate detection using RGB cameras under dark or low-light conditions. Cross-spectral systems must be developed to integrate complementary information from multiple sensor modalities, such as thermal and visible cameras, to improve the robustness of the detections. Second, pedestrian detection models are latency-sensitive. Efficient and easy-to-scale detection models with fewer parameters are highly desirable for real-time applications such as autonomous driving. Third, pedestrian video data provides spatial-temporal correlations of pedestrian movement. It is beneficial to incorporate temporal as well as spatial information to enhance pedestrian detection. This work leverages recent advances in the state space model (Mamba) and proposes a novel Multi-head Hierarchical Patching and Aggregation (MHHPA) structure to extract both fine-grained and coarse-grained information from both RGB and thermal imagery. Experimental results show that the proposed MHHPA is an effective and efficient alternative to a Transformer model for cross-spectral pedestrian detection. Our proposed model also achieves superior performance on small-scale pedestrian detection. The code is available at https://github.com/XiangboGaoBarry/MambaST}{https://github.com/XiangboGaoBarry/MambaST.