# Online Human Action Detection using Joint Classification-Regression Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1604.05633 · HTML: https://mlsanity.com/paper/1604.05633/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Apr 2016.

## Built on by (most cited first, 28 of 28)

- [OadTR: Online Action Detection with Transformers](https://mlsanity.com/paper/2106.11149/) (2021)
- [iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysis](https://mlsanity.com/paper/2107.00285/) (2021)
- [Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detection](https://mlsanity.com/paper/2108.03619/) (2021)
- [Hierarchically Self-Supervised Transformer for Human Skeleton Representation Learning](https://mlsanity.com/paper/2207.09644/) (2022)
- [LAC: Latent Action Composition for Skeleton-based Action Segmentation](https://mlsanity.com/paper/2308.14500/) (2023)
- [Context-Enhanced Memory-Refined Transformer for Online Action Detection](https://mlsanity.com/paper/2503.18359/) (2025)
- [LocATe: End-to-end Localization of Actions in 3D with Transformers](https://mlsanity.com/paper/2203.10719/) (2022)
- [Identity-free Artificial Emotional Intelligence via Micro-Gesture Understanding](https://mlsanity.com/paper/2405.13206/) (2024)
- [Untrimmed Action Anticipation](https://mlsanity.com/paper/2202.04132/) (2022)
- [Video-based Human Action Recognition using Deep Learning: A Review](https://mlsanity.com/paper/2208.03775/) (2022)
- [SimOn: A Simple Framework for Online Temporal Action Localization](https://mlsanity.com/paper/2211.04905/) (2022)
- [Review of Video Predictive Understanding: Early Action Recognition and Future Action Prediction](https://mlsanity.com/paper/2107.05140/) (2021)
- [RNNs, CNNs and Transformers in Human Action Recognition: A Survey and a Hybrid Model](https://mlsanity.com/paper/2407.06162/) (2024)
- [Online recognition of unsegmented actions with hierarchical SOM architecture](https://mlsanity.com/paper/2104.11637/) (2021)
- [Head and eye egocentric gesture recognition for human-robot interaction using eyewear cameras](https://mlsanity.com/paper/2201.11500/) (2022)
- [Centre Stage: Centricity-based Audio-Visual Temporal Action Detection](https://mlsanity.com/paper/2311.16446/) (2023)
- [Live Video Captioning](https://mlsanity.com/paper/2406.14206/) (2024)
- [Refining Action Boundaries for One-stage Detection](https://mlsanity.com/paper/2210.14284/) (2022)
- [Deep learning-based approaches for human motion decoding in smart walkers for rehabilitation](https://mlsanity.com/paper/2301.05575/) (2023)
- [Deep Neural Networks in Video Human Action Recognition: A Review](https://mlsanity.com/paper/2305.15692/) (2023)
- [TS-RGBD Dataset: a Novel Dataset for Theatre Scenes Description for People with Visual Impairments](https://mlsanity.com/paper/2308.01035/) (2023)
- [SFHand: Learning Embodied Manipulation by Streaming Egocentric 3D Hand Forecasting](https://mlsanity.com/paper/2511.18127/) (2025)
- [Real-Time Online Skeleton Extraction and Gesture Recognition on Pepper](https://mlsanity.com/paper/2206.11376/) (2022)
- [Motion-R3: Fast and Accurate Motion Annotation via Representation-based Representativeness Ranking](https://mlsanity.com/paper/2304.01672/) (2023)
- [Online Human Action Detection during Escorting](https://mlsanity.com/paper/2506.23573/) (2025)
- [Accurate online action and gesture recognition system using detectors and Deep SPD Siamese Networks](https://mlsanity.com/paper/2511.05250/) (2025)
- [Improving Viewpoint-Invariance and Temporal Consistency for Action Detection](https://mlsanity.com/paper/2605.22695/) (2026)
- [T-MOR: Learning Motion-Aware Skeleton Representations for Human Action Recognition](https://mlsanity.com/paper/2606.21607/) (2026)

## Abstract (arXiv)

Human action recognition from well-segmented 3D skeleton data has been intensively studied and has been attracting an increasing attention. Online action detection goes one step further and is more challenging, which identifies the action type and localizes the action positions on the fly from the untrimmed stream data. In this paper, we study the problem of online action detection from streaming skeleton data. We propose a multi-task end-to-end Joint Classification-Regression Recurrent Neural Network to better explore the action type and temporal localization information. By employing a joint classification and regression optimization objective, this network is capable of automatically localizing the start and end points of actions more accurately. Specifically, by leveraging the merits of the deep Long Short-Term Memory (LSTM) subnetwork, the proposed model automatically captures the complex long-range temporal dynamics, which naturally avoids the typical sliding window design and thus ensures high computational efficiency. Furthermore, the subtask of regression optimization provides the ability to forecast the action prior to its occurrence. To evaluate our proposed model, we build a large streaming video dataset with annotations. Experimental results on our dataset and the public G3D dataset both demonstrate very promising performance of our scheme.