# SpikeMba: Multi-Modal Spiking Saliency Mamba for Temporal Video Grounding

arXiv: https://arxiv.org/abs/2404.01174 · HTML: https://mlsanity.com/paper/2404.01174/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Video, motion and generative Mamba models, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 31 of its 45 references that are in the map.
- First posted on arXiv 1 Apr 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Video, motion and generative Mamba models](https://mlsanity.com/ssm/video-motion-and-generative-mamba-models/)

## Builds on (most cited first, 30 of 31)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [VMamba: Visual State Space Model](https://mlsanity.com/paper/2401.10166/) (2024)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation](https://mlsanity.com/paper/2401.04722/) (2024)
- [Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks](https://mlsanity.com/paper/2007.05785/) (2020)
- [Deep Residual Learning in Spiking Neural Networks](https://mlsanity.com/paper/2102.04159/) (2021)
- [VideoMamba: State Space Model for Efficient Video Understanding](https://mlsanity.com/paper/2403.06977/) (2024)
- [VM-UNet: Vision Mamba UNet for Medical Image Segmentation](https://mlsanity.com/paper/2402.02491/) (2024)
- [Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language](https://mlsanity.com/paper/1912.03590/) (2019)
- [SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentation](https://mlsanity.com/paper/2401.13560/) (2024)
- [TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval](https://mlsanity.com/paper/2001.09099/) (2020)
- [UniVTG: Towards Unified Video-Language Temporal Grounding](https://mlsanity.com/paper/2307.16715/) (2023)
- [Dense Regression Network for Video Grounding](https://mlsanity.com/paper/2004.03545/) (2020)
- [Query-Dependent Video Representation for Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2303.13874/) (2023)
- [UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2203.12745/) (2022)
- [Knowing Where to Focus: Event-aware Transformer for Video Grounding](https://mlsanity.com/paper/2308.06947/) (2023)
- [Pan-Mamba: Effective pan-sharpening with State Space Model](https://mlsanity.com/paper/2402.12192/) (2024)
- [Less is More: Learning Highlight Detection from Video Duration](https://mlsanity.com/paper/1903.00859/) (2019)
- [Vivim: a Video Vision Mamba for Medical Video Segmentation](https://mlsanity.com/paper/2401.14168/) (2024)
- [VLG-Net: Video-Language Graph Matching Network for Video Grounding](https://mlsanity.com/paper/2011.10132/) (2020)
- [Pruning of Deep Spiking Neural Networks through Gradient Rewiring](https://mlsanity.com/paper/2105.04916/) (2021)
- [MINI-Net: Multiple Instance Ranking Network for Video Highlight Detection](https://mlsanity.com/paper/2007.09833/) (2020)
- [SEENN: Towards Temporal Spiking Early-Exit Neural Networks](https://mlsanity.com/paper/2304.01230/) (2023)
- [Cross-category Video Highlight Detection via Set-based Learning](https://mlsanity.com/paper/2108.11770/) (2021)
- [SpikePoint: An Efficient Point-based Spiking Neural Network for Event Cameras Action Recognition](https://mlsanity.com/paper/2310.07189/) (2023)
- [Scanning Only Once: An End-to-end Framework for Fast Temporal Grounding in Long Videos](https://mlsanity.com/paper/2303.08345/) (2023)
- [Can Shuffling Video Benefit Temporal Bias Problem: A Novel Training Framework for Temporal Grounding](https://mlsanity.com/paper/2207.14698/) (2022)
- [Language-free Training for Zero-shot Video Grounding](https://mlsanity.com/paper/2210.12977/) (2022)

## Built on by (most cited first, 25 of 25)

- [Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges](https://mlsanity.com/paper/2404.16112/) (2024)
- [State Space Model for New-Generation Network Alternative to Transformers: A Survey](https://mlsanity.com/paper/2404.09516/) (2024)
- [Vision Mamba: A Comprehensive Survey and Taxonomy](https://mlsanity.com/paper/2405.04404/) (2024)
- [A Survey on Visual Mamba](https://mlsanity.com/paper/2404.15956/) (2024)
- [Visual Mamba: A Survey and New Outlooks](https://mlsanity.com/paper/2404.18861/) (2024)
- [A Survey of Mamba](https://mlsanity.com/paper/2408.01129/) (2024)
- [The Evolution of Multimodal Model Architectures](https://mlsanity.com/paper/2405.17927/) (2024)
- [Mamba in Vision: A Comprehensive Survey of Techniques and Applications](https://mlsanity.com/paper/2410.03105/) (2024)
- [Snakes and Ladders: Two Steps Up for VideoMamba](https://mlsanity.com/paper/2406.19006/) (2024)
- [CoMamba: Real-time Cooperative Perception Unlocked with State Space Models](https://mlsanity.com/paper/2409.10699/) (2024)
- [Skip Mamba Diffusion for Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2501.07260/) (2025)
- [SHMamba: Structured Hyperbolic State Space Model for Audio-Visual Question Answering](https://mlsanity.com/paper/2406.09833/) (2024)
- [Spiking Variational Graph Representation Inference for Video Summarization](https://mlsanity.com/paper/2508.15389/) (2025)
- [Moment Quantization for Video Temporal Grounding](https://mlsanity.com/paper/2504.02286/) (2025)
- [Sim-DETR: Unlock DETR for Temporal Sentence Grounding](https://mlsanity.com/paper/2509.23867/) (2025)
- [GazeSCRNN: Event-based Near-eye Gaze Tracking using a Spiking Neural Network](https://mlsanity.com/paper/2503.16012/) (2025)
- [MedVSR: Medical Video Super-Resolution with Cross State-Space Propagation](https://mlsanity.com/paper/2509.21265/) (2025)
- [SpikingMamba: Towards Energy-Efficient Large Language Models via Knowledge Distillation from Mamba](https://mlsanity.com/paper/2510.04595/) (2025)
- [CymbaDiff: Structured Spatial Diffusion for Sketch-based 3D Semantic Urban Scene Generation](https://mlsanity.com/paper/2510.13245/) (2025)
- [Technologies on Effectiveness and Efficiency: A Survey of State Spaces Models](https://mlsanity.com/paper/2503.11224/) (2025)
- [Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning](https://mlsanity.com/paper/2505.19938/) (2025)
- [HRSAM: Efficient Interactive Segmentation in High-Resolution Images](https://mlsanity.com/paper/2407.02109/) (2024)
- [Multi-granular body modeling with Redundancy-Free Spatiotemporal Fusion for Text-Driven Motion Generation](https://mlsanity.com/paper/2503.06897/) (2025)
- [Vision SmolMamba: Spike-Guided Token Pruning for Energy-Efficient Spiking State-Space Vision Models](https://mlsanity.com/paper/2604.25570/) (2026)
- [GenSpan: Generation-Calibrated Motion Span Priors for Multi-Verb Video Corpus Moment Retrieval](https://mlsanity.com/paper/2603.22121/) (2026)

## Abstract (arXiv)

Temporal video grounding (TVG) is a critical task in video content understanding, requiring precise alignment between video content and natural language instructions. Despite significant advancements, existing methods face challenges in managing confidence bias towards salient objects and capturing long-term dependencies in video sequences. To address these issues, we introduce SpikeMba: a multi-modal spiking saliency mamba for temporal video grounding. Our approach integrates Spiking Neural Networks (SNNs) with state space models (SSMs) to leverage their unique advantages in handling different aspects of the task. Specifically, we use SNNs to develop a spiking saliency detector that generates the proposal set. The detector emits spike signals when the input signal exceeds a predefined threshold, resulting in a dynamic and binary saliency proposal set. To enhance the model's capability to retain and infer contextual information, we introduce relevant slots which learnable tensors that encode prior knowledge. These slots work with the contextual moment reasoner to maintain a balance between preserving contextual information and exploring semantic relevance dynamically. The SSMs facilitate selective information propagation, addressing the challenge of long-term dependency in video content. By combining SNNs for proposal generation and SSMs for effective contextual reasoning, SpikeMba addresses confidence bias and long-term dependencies, thereby significantly enhancing fine-grained multimodal relationship capture. Our experiments demonstrate the effectiveness of SpikeMba, which consistently outperforms state-of-the-art methods across mainstream benchmarks.