# Differential Recurrent Neural Networks for Action Recognition

arXiv: https://arxiv.org/abs/1504.06678 · HTML: https://mlsanity.com/paper/1504.06678/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2015.

## Built on by (most cited first, 30 of 39)

- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [Selective Structured State-Spaces for Long-Form Video Understanding](https://mlsanity.com/paper/2303.14526/) (2023)
- [Hierarchically Decomposed Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2208.10741/) (2022)
- [iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysis](https://mlsanity.com/paper/2107.00285/) (2021)
- [Learning Discriminative Representations for Skeleton Based Action Recognition](https://mlsanity.com/paper/2303.03729/) (2023)
- [LLMs are Good Action Recognizers](https://mlsanity.com/paper/2404.00532/) (2024)
- [Language Knowledge-Assisted Representation Learning for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2305.12398/) (2023)
- [COMISR: Compression-Informed Video Super-Resolution](https://mlsanity.com/paper/2105.01237/) (2021)
- [No frame left behind: Full Video Action Recognition](https://mlsanity.com/paper/2103.15395/) (2021)
- [Skeleton-Based Mutually Assisted Interacted Object Localization and Human Action Recognition](https://mlsanity.com/paper/2110.14994/) (2021)
- [Skeleton-based Action Recognition via Temporal-Channel Aggregation](https://mlsanity.com/paper/2205.15936/) (2022)
- [Leveraging Spatio-Temporal Dependency for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2212.04761/) (2022)
- [Simba: Mamba augmented U-ShiftGCN for Skeletal Action Recognition in Videos](https://mlsanity.com/paper/2404.07645/) (2024)
- [Pyramid Self-attention Polymerization Learning for Semi-supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2302.02327/) (2023)
- [Graph-based Multi-ODE Neural Networks for Spatio-Temporal Traffic Forecasting](https://mlsanity.com/paper/2305.18687/) (2023)
- [Adaptive Hyper-Graph Convolution Network for Skeleton-based Human Action Recognition with Virtual Connections](https://mlsanity.com/paper/2411.14796/) (2024)
- [Explore Human Parsing Modality for Action Recognition](https://mlsanity.com/paper/2401.02138/) (2024)
- [Identity-free Artificial Emotional Intelligence via Micro-Gesture Understanding](https://mlsanity.com/paper/2405.13206/) (2024)
- [Action Recognition with Multi-stream Motion Modeling and Mutual Information Maximization](https://mlsanity.com/paper/2306.07576/) (2023)
- [Skeleton Sequence and RGB Frame Based Multi-Modality Feature Fusion Network for Action Recognition](https://mlsanity.com/paper/2202.11374/) (2022)
- [3D Skeleton-Based Action Recognition: A Review](https://mlsanity.com/paper/2506.00915/) (2025)
- [MK-SGN: A Spiking Graph Convolutional Network with Multimodal Fusion and Knowledge Distillation for Skeleton-based Action Recognition](https://mlsanity.com/paper/2404.10210/) (2024)
- [Hierarchical growing grid networks for skeleton based action recognition](https://mlsanity.com/paper/2104.11165/) (2021)
- [Skeleton-based Human Action Recognition via Convolutional Neural Networks (CNN)](https://mlsanity.com/paper/2301.13360/) (2023)
- [Attention Prompt Tuning: Parameter-efficient Adaptation of Pre-trained Models for Spatiotemporal Modeling](https://mlsanity.com/paper/2403.06978/) (2024)
- [Deep video representation learning: a survey](https://mlsanity.com/paper/2405.06574/) (2024)
- [Transformers in Action Recognition: A Review on Temporal Modeling](https://mlsanity.com/paper/2302.01921/) (2022)
- [Action Recognition in Real-World Ambient Assisted Living Environment](https://mlsanity.com/paper/2503.23214/) (2025)
- [Action in Mind: A Neural Network Approach to Action Recognition and Segmentation](https://mlsanity.com/paper/2104.14870/) (2021)
- [Spiking Two-Stream Methods with Unsupervised STDP-based Learning for Action Recognition](https://mlsanity.com/paper/2306.13783/) (2023)

## Abstract (arXiv)

The long short-term memory (LSTM) neural network is capable of processing complex sequential information since it utilizes special gating schemes for learning representations from long input sequences. It has the potential to model any sequential time-series data, where the current hidden state has to be considered in the context of the past hidden states. This property makes LSTM an ideal choice to learn the complex dynamics of various actions. Unfortunately, the conventional LSTMs do not consider the impact of spatio-temporal dynamics corresponding to the given salient motion patterns, when they gate the information that ought to be memorized through time. To address this problem, we propose a differential gating scheme for the LSTM neural network, which emphasizes on the change in information gain caused by the salient motions between the successive frames. This change in information gain is quantified by Derivative of States (DoS), and thus the proposed LSTM model is termed as differential Recurrent Neural Network (dRNN). We demonstrate the effectiveness of the proposed model by automatically recognizing actions from the real-world 2D and 3D human action datasets. Our study is one of the first works towards demonstrating the potential of learning complex time-series representations via high-order derivatives of states.