# Spiking Two-Stream Methods with Unsupervised STDP-based Learning for Action Recognition

arXiv: https://arxiv.org/abs/2306.13783 · HTML: https://mlsanity.com/paper/2306.13783/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 71 references that are in the map.
- First posted on arXiv 23 Jun 2023.

## Builds on (most cited first, 28 of 28)

- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks](https://mlsanity.com/paper/1706.02609/) (2017)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [Deep Learning in Spiking Neural Networks](https://mlsanity.com/paper/1804.08150/) (2018)
- [OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks](https://mlsanity.com/paper/1312.6229/) (2013)
- [Deep Learning for Sensor-based Activity Recognition: A Survey](https://mlsanity.com/paper/1707.03502/) (2017)
- [View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data](https://mlsanity.com/paper/1703.08274/) (2017)
- [Deep, Convolutional, and Recurrent Models for Human Activity Recognition using Wearables](https://mlsanity.com/paper/1604.08880/) (2016)
- [A Survey of Neuromorphic Computing and Neural Networks in Hardware](https://mlsanity.com/paper/1705.06963/) (2017)
- [Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks](https://mlsanity.com/paper/1704.02581/) (2017)
- [The BrainScaleS-2 accelerated neuromorphic system with hybrid plasticity](https://mlsanity.com/paper/2201.11063/) (2022)
- [Differential Recurrent Neural Networks for Action Recognition](https://mlsanity.com/paper/1504.06678/) (2015)
- [Unsupervised Learning from Video with Deep Neural Embeddings](https://mlsanity.com/paper/1905.11954/) (2019)
- [SpikeMS: Deep Spiking Neural Network for Motion Segmentation](https://mlsanity.com/paper/2105.06562/) (2021)
- [Two-Stream 3D Convolutional Neural Network for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1705.08106/) (2017)
- [On-Off Center-Surround Receptive Fields for Accurate and Robust Image Classification](https://mlsanity.com/paper/2106.07091/) (2021)
- [Multi-layered Spiking Neural Network with Target Timestamp Threshold Adaptation and STDP](https://mlsanity.com/paper/1904.01908/) (2019)
- [Unsupervised Visual Feature Learning with Spike-timing-dependent Plasticity: How Far are we from Traditional Feature Learning Approaches?](https://mlsanity.com/paper/1901.04392/) (2019)
- [Bio-Inspired Spiking Convolutional Neural Network using Layer-wise Sparse Coding and STDP Learning](https://mlsanity.com/paper/1611.03000/) (2016)
- [Evaluating Two-Stream CNN for Video Classification](https://mlsanity.com/paper/1504.01920/) (2015)
- [Efficient Two-Stream Motion and Appearance 3D CNNs for Video Classification](https://mlsanity.com/paper/1608.08851/) (2016)
- [2D versus 3D Convolutional Spiking Neural Networks Trained with Unsupervised STDP for Human Action Recognition](https://mlsanity.com/paper/2205.13474/) (2022)
- [A Study On the Effects of Pre-processing On Spatio-temporal Action Recognition Using Spiking Neural Networks Trained with STDP](https://mlsanity.com/paper/2105.14740/) (2021)

## Built on by (most cited first, 1 of 1)

- [ReSpike: Residual Frames-based Hybrid Spiking Neural Networks for Efficient Action Recognition](https://mlsanity.com/paper/2409.01564/) (2024)

## Abstract (arXiv)

Video analysis is a computer vision task that is useful for many applications like surveillance, human-machine interaction, and autonomous vehicles. Deep Convolutional Neural Networks (CNNs) are currently the state-of-the-art methods for video analysis. However they have high computational costs, and need a large amount of labeled data for training. In this paper, we use Convolutional Spiking Neural Networks (CSNNs) trained with the unsupervised Spike Timing-Dependent Plasticity (STDP) learning rule for action classification. These networks represent the information using asynchronous low-energy spikes. This allows the network to be more energy efficient and neuromorphic hardware-friendly. However, the behaviour of CSNNs is not studied enough with spatio-temporal computer vision models. Therefore, we explore transposing two-stream neural networks into the spiking domain. Implementing this model with unsupervised STDP-based CSNNs allows us to further study the performance of these networks with video analysis. In this work, we show that two-stream CSNNs can successfully extract spatio-temporal information from videos despite using limited training data, and that the spiking spatial and temporal streams are complementary. We also show that using a spatio-temporal stream within a spiking STDP-based two-stream architecture leads to information redundancy and does not improve the performance.