# Procedural Generation of Videos to Train Deep Action Recognition Networks

arXiv: https://arxiv.org/abs/1612.00881 · HTML: https://mlsanity.com/paper/1612.00881/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2016.

## Built on by (most cited first, 30 of 31)

- [A survey of synthetic data augmentation methods in computer vision](https://mlsanity.com/paper/2403.10075/) (2024)
- [Going Beyond Nouns With Vision & Language Models Using Synthetic Data](https://mlsanity.com/paper/2303.17590/) (2023)
- [GAIA: Rethinking Action Quality Assessment for AI-Generated Videos](https://mlsanity.com/paper/2406.06087/) (2024)
- [Synthetic Data in Human Analysis: A Survey](https://mlsanity.com/paper/2208.09191/) (2022)
- [Task2Sim : Towards Effective Pre-training and Transfer from Synthetic Data](https://mlsanity.com/paper/2112.00054/) (2021)
- [EgoEnv: Human-centric environment representations from egocentric video](https://mlsanity.com/paper/2207.11365/) (2022)
- [Synthetic-to-Real Domain Adaptation for Action Recognition: A Dataset and Baseline Performances](https://mlsanity.com/paper/2303.10280/) (2023)
- [Deep Unsupervised Key Frame Extraction for Efficient Video Classification](https://mlsanity.com/paper/2211.06742/) (2022)
- [PromptonomyViT: Multi-Task Prompt Learning Improves Video Transformers using Synthetic Scene Data](https://mlsanity.com/paper/2212.04821/) (2022)
- [A Large-scale Study of Spatiotemporal Representation Learning with a New Benchmark on Action Recognition](https://mlsanity.com/paper/2303.13505/) (2023)
- [PeopleSansPeople: A Synthetic Data Generator for Human-Centric Computer Vision](https://mlsanity.com/paper/2112.09290/) (2021)
- [M3Act: Learning from Synthetic Human Group Activities](https://mlsanity.com/paper/2306.16772/) (2023)
- [Learning Human Action Recognition Representations Without Real Humans](https://mlsanity.com/paper/2311.06231/) (2023)
- [A Procedural World Generation Framework for Systematic Evaluation of Continual Learning](https://mlsanity.com/paper/2106.02585/) (2021)
- [SVLTA: Benchmarking Vision-Language Temporal Alignment via Synthetic Video Situation](https://mlsanity.com/paper/2504.05925/) (2025)
- [Training Deep Learning Algorithms on Synthetic Forest Images for Tree Detection](https://mlsanity.com/paper/2210.04104/) (2022)
- [Activity Recognition on Avatar-Anonymized Datasets with Masked Differential Privacy](https://mlsanity.com/paper/2410.17098/) (2024)
- [Multimodal Generation of Novel Action Appearances for Synthetic-to-Real Recognition of Activities of Daily Living](https://mlsanity.com/paper/2208.01910/) (2022)
- [An Animation-based Augmentation Approach for Action Recognition from Discontinuous Video](https://mlsanity.com/paper/2404.06741/) (2024)
- [Data Collection-free Masked Video Modeling](https://mlsanity.com/paper/2409.06665/) (2024)
- [Nuisance-Label Supervision: Robustness Improvement by Free Labels](https://mlsanity.com/paper/2110.07118/) (2021)
- [Bent & Broken Bicycles: Leveraging synthetic data for damaged object re-identification](https://mlsanity.com/paper/2304.07883/) (2023)
- [CrashCar101: Procedural Generation for Damage Assessment](https://mlsanity.com/paper/2311.06536/) (2023)
- [Designing Multi-Robot Ground Video Sensemaking with Public Safety Professionals](https://mlsanity.com/paper/2602.08882/) (2026)
- [Compositional Benchmark Synthesis for Hierarchical Human Action Recognition](https://mlsanity.com/paper/2608.10765/) (2026)
- [Unsupervised Super-Resolution of Satellite Imagery for High Fidelity Material Label Transfer](https://mlsanity.com/paper/2105.07322/) (2021)
- [Efficient Realistic Data Generation Framework leveraging Deep Learning-based Human Digitization](https://mlsanity.com/paper/2106.15409/) (2021)
- [Learning Dense Correspondence from Synthetic Environments](https://mlsanity.com/paper/2203.12919/) (2022)
- [SynthoGestures: A Novel Framework for Synthetic Dynamic Hand Gesture Generation for Driving Scenarios](https://mlsanity.com/paper/2309.04421/) (2023)
- [A Unified Simulation Framework for Visual and Behavioral Fidelity in Crowd Analysis](https://mlsanity.com/paper/2312.02613/) (2023)

## Abstract (arXiv)

Deep learning for human action recognition in videos is making significant progress, but is slowed down by its dependency on expensive manual labeling of large video collections. In this work, we investigate the generation of synthetic training data for action recognition, as it has recently shown promising results for a variety of other computer vision tasks. We propose an interpretable parametric generative model of human action videos that relies on procedural generation and other computer graphics techniques of modern game engines. We generate a diverse, realistic, and physically plausible dataset of human action videos, called PHAV for "Procedural Human Action Videos". It contains a total of 39,982 videos, with more than 1,000 examples for each action of 35 categories. Our approach is not limited to existing motion capture sequences, and we procedurally define 14 synthetic actions. We introduce a deep multi-task representation learning architecture to mix synthetic and real videos, even if the action categories differ. Our experiments on the UCF101 and HMDB51 benchmarks suggest that combining our large set of synthetic videos with small real-world datasets can boost recognition performance, significantly outperforming fine-tuning state-of-the-art unsupervised generative models of videos.