# RGB-D-based Action Recognition Datasets: A Survey

arXiv: https://arxiv.org/abs/1601.05511 · HTML: https://mlsanity.com/paper/1601.05511/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jan 2016.

## Built on by (most cited first, 14 of 14)

- [Deep Reinforcement Learning in Computer Vision: A Comprehensive Survey](https://mlsanity.com/paper/2108.11510/) (2021)
- [A Survey on Video Action Recognition in Sports: Datasets, Methods and Applications](https://mlsanity.com/paper/2206.01038/) (2022)
- [Simitate: A Hybrid Imitation Learning Benchmark](https://mlsanity.com/paper/1905.06002/) (2019)
- [Video-based Human Action Recognition using Deep Learning: A Review](https://mlsanity.com/paper/2208.03775/) (2022)
- [From CNNs to Transformers in Multimodal Human Action Recognition: A Survey](https://mlsanity.com/paper/2405.15813/) (2024)
- [A Central Difference Graph Convolutional Operator for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2111.06995/) (2021)
- [Video Violence Recognition and Localization Using a Semi-Supervised Hard Attention Model](https://mlsanity.com/paper/2202.02212/) (2022)
- [Learning Mutual Excitation for Hand-to-Hand and Human-to-Human Interaction Recognition](https://mlsanity.com/paper/2402.02431/) (2024)
- [EgoDTM: Towards 3D-Aware Egocentric Video-Language Pretraining](https://mlsanity.com/paper/2503.15470/) (2025)
- [GAMUS: A Geometry-aware Multi-modal Semantic Segmentation Benchmark for Remote Sensing Data](https://mlsanity.com/paper/2305.14914/) (2023)
- [Transformers in Action Recognition: A Review on Temporal Modeling](https://mlsanity.com/paper/2302.01921/) (2022)
- [Multi-person 3D pose estimation from unlabelled data](https://mlsanity.com/paper/2212.08731/) (2022)
- [Adversarial Interaction Attack: Fooling AI to Misinterpret Human Intentions](https://mlsanity.com/paper/2101.06704/) (2021)
- [E2E-GNet: An End-to-End Skeleton-based Geometric Deep Neural Network for Human Motion Recognition](https://mlsanity.com/paper/2603.02477/) (2026)

## Abstract (arXiv)

Human action recognition from RGB-D (Red, Green, Blue and Depth) data has attracted increasing attention since the first work reported in 2010. Over this period, many benchmark datasets have been created to facilitate the development and evaluation of new algorithms. This raises the question of which dataset to select and how to use it in providing a fair and objective comparative evaluation against state-of-the-art methods. To address this issue, this paper provides a comprehensive review of the most commonly used action recognition related RGB-D video datasets, including 27 single-view datasets, 10 multi-view datasets, and 7 multi-person datasets. The detailed information and analysis of these datasets is a useful resource in guiding insightful selection of datasets for future research. In addition, the issues with current algorithm evaluation vis-'a-vis limitations of the available datasets and evaluation protocols are also highlighted; resulting in a number of recommendations for collection of new datasets and use of evaluation protocols.