# Two-Stream 3D Convolutional Neural Network for Skeleton-Based Action Recognition

arXiv: https://arxiv.org/abs/1705.08106 · HTML: https://mlsanity.com/paper/1705.08106/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 May 2017.

## Built on by (most cited first, 23 of 23)

- [Revisiting Skeleton-based Action Recognition](https://mlsanity.com/paper/2104.13586/) (2021)
- [IGFormer: Interaction Graph Transformer for Skeleton-based Human Interaction Recognition](https://mlsanity.com/paper/2207.12100/) (2022)
- [FLAG3D: A 3D Fitness Activity Dataset with Language Instruction](https://mlsanity.com/paper/2212.04638/) (2022)
- [Skeleton-Based Mutually Assisted Interacted Object Localization and Human Action Recognition](https://mlsanity.com/paper/2110.14994/) (2021)
- [Skeleton-based Action Recognition via Temporal-Channel Aggregation](https://mlsanity.com/paper/2205.15936/) (2022)
- [Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognition](https://mlsanity.com/paper/2407.15706/) (2024)
- [Dynamic Hypergraph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2112.10570/) (2021)
- [STAR: Sparse Transformer-based Action Recognition](https://mlsanity.com/paper/2107.07089/) (2021)
- [Action Segmentation Using 2D Skeleton Heatmaps and Multi-Modality Fusion](https://mlsanity.com/paper/2309.06462/) (2023)
- [Human Pose-based Estimation, Tracking and Action Recognition with Deep Learning: A Survey](https://mlsanity.com/paper/2310.13039/) (2023)
- [3D Skeleton-Based Action Recognition: A Review](https://mlsanity.com/paper/2506.00915/) (2025)
- [An Explainable Non-local Network for COVID-19 Diagnosis](https://mlsanity.com/paper/2408.04300/) (2024)
- [Transformers in Action Recognition: A Review on Temporal Modeling](https://mlsanity.com/paper/2302.01921/) (2022)
- [Learning by Aligning 2D Skeleton Sequences and Multi-Modality Fusion](https://mlsanity.com/paper/2305.19480/) (2023)
- [3DYoga90: A Hierarchical Video Dataset for Yoga Pose Understanding](https://mlsanity.com/paper/2310.10131/) (2023)
- [REGINA - Reasoning Graph Convolutional Networks in Human Action Recognition](https://mlsanity.com/paper/2105.06711/) (2021)
- [Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions](https://mlsanity.com/paper/2108.13055/) (2021)
- [A Comprehensive Methodological Survey of Human Activity Recognition Across Divers Data Modalities](https://mlsanity.com/paper/2409.09678/) (2024)
- [Spiking Two-Stream Methods with Unsupervised STDP-based Learning for Action Recognition](https://mlsanity.com/paper/2306.13783/) (2023)
- [Transformer-based Fusion of 2D-pose and Spatio-temporal Embeddings for Distracted Driver Action Recognition](https://mlsanity.com/paper/2403.06577/) (2024)
- [A New Adjacency Matrix Configuration in GCN-based Models for Skeleton-based Action Recognition](https://mlsanity.com/paper/2206.14344/) (2022)
- [Multi-Dimensional Refinement Graph Convolutional Network with Robust Decouple Loss for Fine-Grained Skeleton-Based Action Recognition](https://mlsanity.com/paper/2306.15321/) (2023)
- [SkeFi: Cross-Modal Knowledge Transfer for Wireless Skeleton-Based Action Recognition](https://mlsanity.com/paper/2601.12432/) (2026)

## Abstract (arXiv)

It remains a challenge to efficiently extract spatialtemporal information from skeleton sequences for 3D human action recognition. Although most recent action recognition methods are based on Recurrent Neural Networks which present outstanding performance, one of the shortcomings of these methods is the tendency to overemphasize the temporal information. Since 3D convolutional neural network(3D CNN) is a powerful tool to simultaneously learn features from both spatial and temporal dimensions through capturing the correlations between three dimensional signals, this paper proposes a novel two-stream model using 3D CNN. To our best knowledge, this is the first application of 3D CNN in skeleton-based action recognition. Our method consists of three stages. First, skeleton joints are mapped into a 3D coordinate space and then encoding the spatial and temporal information, respectively. Second, 3D CNN models are seperately adopted to extract deep features from two streams. Third, to enhance the ability of deep features to capture global relationships, we extend every stream into multitemporal version. Extensive experiments on the SmartHome dataset and the large-scale NTU RGB-D dataset demonstrate that our method outperforms most of RNN-based methods, which verify the complementary property between spatial and temporal information and the robustness to noise.