mlsanityData to 14 Oct 2026

ReSpike: Residual Frames-based Hybrid Spiking Neural Networks for Efficient Action Recognition

arXiv 2409.01564 ↗First posted 3 Sep 2024cs.CV cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Spiking Neural Networks (SNNs) have emerged as a compelling, energy-efficient alternative to traditional Artificial Neural Networks (ANNs) for static image tasks such as image classification and segmentation. However, in the more complex video classification domain, SNN-based methods fall considerably short of ANN-based benchmarks due to the challenges in processing dense frame sequences. To bridge this gap, we propose ReSpike, a hybrid framework that synergizes the strengths of ANNs and SNNs to tackle action recognition tasks with high accuracy and low energy cost. By decomposing film clips into spatial and temporal components, i.e., RGB image Key Frames and event-like Residual Frames, ReSpike leverages ANN for learning spatial information and SNN for learning temporal information. In addition, we propose a multi-scale cross-attention mechanism for effective feature fusion. Compared to state-of-the-art SNN baselines, our ReSpike hybrid architecture demonstrates significant performance improvements (e.g., >30% absolute accuracy improvement on HMDB-51, UCF-101, and Kinetics-400). Furthermore, ReSpike achieves comparable performance with prior ANN approaches while bringing better accuracy-energy tradeoff.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 57 of its 86 references that are in the map. First posted on arXiv 3 Sep 2024.

Builds on

57 of 86 references.

References · 57

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2017Mask R-CNNcited by 7,277
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  9. 2014Going Deeper with Convolutionscited by 4,241
  10. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  11. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  12. 2016Identity Mappings in Deep Residual Networkscited by 1,965
38 more
  1. 2017The Kinetics Human Action Video Datasetcited by 1,909
  2. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  3. 2021Multiscale Vision Transformerscited by 897
  4. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
  5. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 626
  6. 2017Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networkscited by 406
  7. 2020Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networkscited by 345
  8. 2021Deep Residual Learning in Spiking Neural Networkscited by 331
  9. 2020Going Deeper With Directly-Trained Larger Spiking Neural Networkscited by 278
  10. 2022Spikformer: When Spiking Neural Network Meets Transformercited by 271
  11. 2018Deep Learning in Spiking Neural Networkscited by 263
  12. 2018Direct Training for Spiking Neural Networks: Faster, Larger, Bettercited by 233
  13. 2016Training Deep Spiking Neural Networks using Backpropagationcited by 205
  14. 2022Temporal Efficient Training of Spiking Neural Network via Gradient Re-weightingcited by 198
  15. 2023Spike-driven Transformercited by 146
  16. 2020Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagationcited by 144
  17. 2021Temporal-wise Attention Spiking Neural Networks for Event Streams Classificationcited by 106
  18. 2020Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networkscited by 98
  19. 2016Spatiotemporal Residual Networks for Video Action Recognitioncited by 91
  20. 2020Revisiting Batch Normalization for Training Low-latency Deep Spiking Neural Networks from Scratchcited by 80
  21. 2021Advancing Spiking Neural Networks towards Deep Residual Learningcited by 79
  22. 2023Deep Directly-Trained Spiking Neural Networks for Object Detectioncited by 77
  23. 2020DIET-SNN: Direct Input Encoding With Leakage and Threshold Optimization in Deep Spiking Neural Networkscited by 71
  24. 2022Event-based Video Reconstruction via Potential-assisted Spiking Neural Networkcited by 59
  25. 2021Beyond Classification: Directly Training Spiking Neural Networks for Semantic Segmentationcited by 55
  26. 2017ConvNet Architecture Search for Spatiotemporal Feature Learningcited by 40
  27. 2022TCJA-SNN: Temporal-Channel Joint Attention for Spiking Neural Networkscited by 35
  28. 2023Membrane Potential Batch Normalization for Spiking Neural Networkscited by 35
  29. 2020Brain-inspired global-local learning incorporated with neuromorphic computingcited by 28
  30. 2021Visual Explanations from Spiking Neural Networks using Interspike Intervalscited by 27
  31. 2021Fusion-FlowNet: Energy-Efficient Optical Flow Estimation using Sensor Fusion and Deep Fused Spiking-Analog Network Architecturescited by 25
  32. 2022Rate Coding or Direct Coding: Which One is Better for Accurate, Robust, and Energy-efficient Spiking Neural Networks?cited by 25
  33. 2022Exploring Temporal Information Dynamics in Spiking Neural Networkscited by 24
  34. 2020Convolutional Spiking Neural Networks for Spatio-Temporal Feature Extractioncited by 23
  35. 2024EAS-SNN: End-to-End Adaptive Sampling and Representation for Event-based Detection with Recurrent Spiking Neural Networkscited by 16
  36. 2023A Hybrid ANN-SNN Architecture for Low-Power and Low-Latency Visual Perceptioncited by 13
  37. 2023DeViT: Decomposing Vision Transformers for Collaborative Inference in Edge Devicescited by 13
  38. 2023Uncovering the Representation of Spiking Neural Networks Trained with Surrogate Gradientcited by 11

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Cross Knowledge Distillation between Artificial and Spiking Neural Networkscited by 4
  2. 2026Fire on Motion: Optimizing Video Pass-bands for Efficient Spiking Action Recognitioncited by 1
  3. 2025SPACT18: Spiking Human Action Recognition Benchmark Dataset with Complementary RGB and Thermal Modalitiescited by 0