# PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding

arXiv: https://arxiv.org/abs/1703.07475 · HTML: https://mlsanity.com/paper/1703.07475/

## Key facts

- Cited by 101 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Mar 2017.

## Built on by (most cited first, 30 of 101)

- [MotionBERT: A Unified Perspective on Learning Human Motion Representations](https://mlsanity.com/paper/2210.06551/) (2022)
- [UAV-Human: A Large Benchmark for Human Behavior Understanding with Unmanned Aerial Vehicles](https://mlsanity.com/paper/2104.00946/) (2021)
- [Skeleton-Contrastive 3D Action Representation Learning](https://mlsanity.com/paper/2108.03656/) (2021)
- [MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models](https://mlsanity.com/paper/2408.02718/) (2024)
- [Home Action Genome: Cooperative Compositional Action Understanding](https://mlsanity.com/paper/2105.05226/) (2021)
- [HYperbolic Self-Paced Learning for Self-Supervised Skeleton-based Action Representations](https://mlsanity.com/paper/2303.06242/) (2023)
- [Masked Motion Predictors are Strong 3D Action Representation Learners](https://mlsanity.com/paper/2308.07092/) (2023)
- [CMD: Self-supervised 3D Action Representation Learning with Cross-modal Mutual Distillation](https://mlsanity.com/paper/2208.12448/) (2022)
- [Hierarchically Self-Supervised Transformer for Human Skeleton Representation Learning](https://mlsanity.com/paper/2207.09644/) (2022)
- [DVANet: Disentangling View and Action Features for Multi-View Action Recognition](https://mlsanity.com/paper/2312.05719/) (2023)
- [Fine-Grained Side Information Guided Dual-Prompts for Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2404.07487/) (2024)
- [SA-DVAE: Improving Zero-Shot Skeleton-Based Action Recognition by Disentangled Variational Autoencoders](https://mlsanity.com/paper/2407.13460/) (2024)
- [Physical Adversarial Attacks for Surveillance: A Survey](https://mlsanity.com/paper/2305.01074/) (2023)
- [LAC: Latent Action Composition for Skeleton-based Action Segmentation](https://mlsanity.com/paper/2308.14500/) (2023)
- [Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2411.11288/) (2024)
- [A Survey on RGB-D Datasets](https://mlsanity.com/paper/2201.05761/) (2022)
- [SCD-Net: Spatiotemporal Clues Disentanglement Network for Self-supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2309.05834/) (2023)
- [Vision-Language Meets the Skeleton: Progressively Distillation with Cross-Modal Knowledge for 3D Action Representation Learning](https://mlsanity.com/paper/2405.20606/) (2024)
- [Human Activity Recognition Using Tools of Convolutional Neural Networks: A State of the Art Review, Data Sets, Challenges and Future Prospects](https://mlsanity.com/paper/2202.03274/) (2022)
- [ANIM: Accurate Neural Implicit Model for Human Reconstruction from a single RGB-D image](https://mlsanity.com/paper/2403.10357/) (2024)
- [Markerless Multi-view 3D Human Pose Estimation: a survey](https://mlsanity.com/paper/2407.03817/) (2024)
- [Computer Vision for Clinical Gait Analysis: A Gait Abnormality Video Dataset](https://mlsanity.com/paper/2407.04190/) (2024)
- [Bridging the Skeleton-Text Modality Gap: Diffusion-Powered Modality Alignment for Zero-shot Skeleton-based Action Recognition](https://mlsanity.com/paper/2411.10745/) (2024)
- [Bootstrapped Representation Learning for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2202.02232/) (2022)
- [Activity-Biometrics: Person Identification from Daily Activities](https://mlsanity.com/paper/2403.17360/) (2024)
- [From CNNs to Transformers in Multimodal Human Action Recognition: A Survey](https://mlsanity.com/paper/2405.15813/) (2024)
- [Towards Physical World Backdoor Attacks against Skeleton Action Recognition](https://mlsanity.com/paper/2408.08671/) (2024)
- [DMMG: Dual Min-Max Games for Self-Supervised Skeleton-Based Action Recognition](https://mlsanity.com/paper/2302.12007/) (2023)
- [Focalized Contrastive View-invariant Learning for Self-supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2304.00858/) (2023)
- [Human Pose-based Estimation, Tracking and Action Recognition with Deep Learning: A Survey](https://mlsanity.com/paper/2310.13039/) (2023)

## Abstract (arXiv)

Despite the fact that many 3D human activity benchmarks being proposed, most existing action datasets focus on the action recognition tasks for the segmented videos. There is a lack of standard large-scale benchmarks, especially for current popular data-hungry deep learning based methods. In this paper, we introduce a new large scale benchmark (PKU-MMD) for continuous multi-modality 3D human action understanding and cover a wide range of complex human activities with well annotated information. PKU-MMD contains 1076 long video sequences in 51 action categories, performed by 66 subjects in three camera views. It contains almost 20,000 action instances and 5.4 million frames in total. Our dataset also provides multi-modality data sources, including RGB, depth, Infrared Radiation and Skeleton. With different modalities, we conduct extensive experiments on our dataset in terms of two scenarios and evaluate different methods by various metrics, including a new proposed evaluation protocol 2D-AP. We believe this large-scale dataset will benefit future researches on action detection for the community.