# Action Recognition with Multi-stream Motion Modeling and Mutual Information Maximization

arXiv: https://arxiv.org/abs/2306.07576 · HTML: https://mlsanity.com/paper/2306.07576/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 59 references that are in the map.
- First posted on arXiv 13 Jun 2023.

## Builds on (most cited first, 30 of 30)

- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1910.03151/) (2019)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information](https://mlsanity.com/paper/2006.12013/) (2020)
- [Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/1904.12659/) (2019)
- [An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1902.09130/) (2019)
- [Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition](https://mlsanity.com/paper/1607.07043/) (2016)
- [Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN](https://mlsanity.com/paper/1803.04831/) (2018)
- [Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition](https://mlsanity.com/paper/1904.01189/) (2019)
- [Cross-view Action Modeling, Learning and Recognition](https://mlsanity.com/paper/1405.2941/) (2014)
- [Constructing Stronger and Faster Baselines for Skeleton-based Action Recognition](https://mlsanity.com/paper/2106.15125/) (2021)
- [Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition](https://mlsanity.com/paper/2007.14690/) (2020)
- [The Devil is in the Details: Delving into Unbiased Data Processing for Human Pose Estimation](https://mlsanity.com/paper/1911.07524/) (2019)
- [Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2206.13028/) (2022)
- [Skeleton-Based Human Action Recognition with Global Context-Aware Attention LSTM Networks](https://mlsanity.com/paper/1707.05740/) (2017)
- [Farewell to Mutual Information: Variational Distillation for Cross-Modal Person Re-Identification](https://mlsanity.com/paper/2104.02862/) (2021)
- [Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning](https://mlsanity.com/paper/1805.02335/) (2018)
- [Differential Recurrent Neural Networks for Action Recognition](https://mlsanity.com/paper/1504.06678/) (2015)
- [Topology-aware Convolutional Neural Network for Efficient Skeleton-based Action Recognition](https://mlsanity.com/paper/2112.04178/) (2021)
- [Temporal Feature Alignment and Mutual Information Maximization for Video-Based Human Pose Estimation](https://mlsanity.com/paper/2203.15227/) (2022)
- [Aggregated Multi-GANs for Controlled 3D Human Motion Prediction](https://mlsanity.com/paper/2103.09755/) (2021)
- [Eye in the Sky: Real-time Drone Surveillance System (DSS) for Violent Individuals Identification using ScatterNet Hybrid Deep Learning Network](https://mlsanity.com/paper/1806.00746/) (2018)
- [Group Contextualization for Video Recognition](https://mlsanity.com/paper/2203.09694/) (2022)
- [Attention in Attention: Modeling Context Correlation for Efficient Video Classification](https://mlsanity.com/paper/2204.09303/) (2022)

## Built on by (most cited first, 7 of 7)

- [DFIL: Deepfake Incremental Learning by Exploiting Domain-invariant Forgery Clues](https://mlsanity.com/paper/2309.09526/) (2023)
- [3D Skeleton-Based Action Recognition: A Review](https://mlsanity.com/paper/2506.00915/) (2025)
- [Shap-Mix: Shapley Value Guided Mixing for Long-Tailed Skeleton Based Action Recognition](https://mlsanity.com/paper/2407.12312/) (2024)
- [Neighbor Does Matter: Density-Aware Contrastive Learning for Medical Semi-supervised Segmentation](https://mlsanity.com/paper/2412.19871/) (2024)
- [Video Infringement Detection via Feature Disentanglement and Mutual Information Maximization](https://mlsanity.com/paper/2309.06877/) (2023)
- [Toward Modality Gap: Vision Prototype Learning for Weakly-supervised Semantic Segmentation with CLIP](https://mlsanity.com/paper/2412.19650/) (2024)
- [Multi-Grained Feature Pruning for Video-Based Human Pose Estimation](https://mlsanity.com/paper/2503.05365/) (2025)

## Abstract (arXiv)

Action recognition has long been a fundamental and intriguing problem in artificial intelligence. The task is challenging due to the high dimensionality nature of an action, as well as the subtle motion details to be considered. Current state-of-the-art approaches typically learn from articulated motion sequences in the straightforward 3D Euclidean space. However, the vanilla Euclidean space is not efficient for modeling important motion characteristics such as the joint-wise angular acceleration, which reveals the driving force behind the motion. Moreover, current methods typically attend to each channel equally and lack theoretical constrains on extracting task-relevant features from the input. In this paper, we seek to tackle these challenges from three aspects: (1) We propose to incorporate an acceleration representation, explicitly modeling the higher-order variations in motion. (2) We introduce a novel Stream-GCN network equipped with multi-stream components and channel attention, where different representations (i.e., streams) supplement each other towards a more precise action recognition while attention capitalizes on those important channels. (3) We explore feature-level supervision for maximizing the extraction of task-relevant information and formulate this into a mutual information loss. Empirically, our approach sets the new state-of-the-art performance on three benchmark datasets, NTU RGB+D, NTU RGB+D 120, and NW-UCLA. Our code is anonymously released at https://github.com/ActionR-Group/Stream-GCN, hoping to inspire the community.