# Let's Play for Action: Recognizing Activities of Daily Living by Learning from Life Simulation Video Games

arXiv: https://arxiv.org/abs/2107.05617 · HTML: https://mlsanity.com/paper/2107.05617/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 34 references that are in the map.
- First posted on arXiv 12 Jul 2021.

## Builds on (most cited first, 20 of 20)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Action2Motion: Conditioned Generation of 3D Human Motions](https://mlsanity.com/paper/2007.15240/) (2020)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [Playing for Benchmarks](https://mlsanity.com/paper/1709.07322/) (2017)
- [VPN: Learning Video-Pose Embedding for Activities of Daily Living](https://mlsanity.com/paper/2007.03056/) (2020)
- [Synthetic Humans for Action Recognition from Unseen Viewpoints](https://mlsanity.com/paper/1912.04070/) (2019)
- [Toyota Smarthome Untrimmed: Real-World Untrimmed Videos for Activity Detection](https://mlsanity.com/paper/2010.14982/) (2020)
- [Automatic Operating Room Surgical Activity Recognition for Robot-Assisted Surgery](https://mlsanity.com/paper/2006.16166/) (2020)
- [IDDA: a large-scale multi-domain dataset for autonomous driving](https://mlsanity.com/paper/2004.08298/) (2020)
- [ElderSim: A Synthetic Data Generation Platform for Human Action Recognition in Eldercare Applications](https://mlsanity.com/paper/2010.14742/) (2020)
- [ETRI-Activity3D: A Large-Scale RGB-D Dataset for Robots to Recognize Daily Activities of the Elderly](https://mlsanity.com/paper/2003.01920/) (2020)
- [Behavior-Driven Synthesis of Human Dynamics](https://mlsanity.com/paper/2103.04677/) (2021)
- [Free-View, 3D Gaze-Guided, Assistive Robotic System for Activities of Daily Living](https://mlsanity.com/paper/1807.05452/) (2018)

## Built on by (most cited first, 17 of 17)

- [Synthetic Data in Human Analysis: A Survey](https://mlsanity.com/paper/2208.09191/) (2022)
- [Interactive Spatiotemporal Token Attention Network for Skeleton-based General Interactive Action Recognition](https://mlsanity.com/paper/2307.07469/) (2023)
- [A Survey of Human Activity Recognition in Smart Homes Based on IoT Sensors Algorithms: Taxonomies, Challenges, and Opportunities with Deep Learning](https://mlsanity.com/paper/2111.04418/) (2021)
- [A Survey on Deep Learning Techniques for Action Anticipation](https://mlsanity.com/paper/2309.17257/) (2023)
- [ModSelect: Automatic Modality Selection for Synthetic-to-Real Domain Generalization](https://mlsanity.com/paper/2208.09414/) (2022)
- [A Two-stream Hybrid CNN-Transformer Network for Skeleton-based Human Interaction Recognition](https://mlsanity.com/paper/2401.00409/) (2023)
- [Towards Synthetic Data Generation for Improved Pain Recognition in Videos under Patient Constraints](https://mlsanity.com/paper/2409.16382/) (2024)
- [Multimodal Generation of Novel Action Appearances for Synthetic-to-Real Recognition of Activities of Daily Living](https://mlsanity.com/paper/2208.01910/) (2022)
- [Exploring Few-Shot Adaptation for Activity Recognition on Diverse Domains](https://mlsanity.com/paper/2305.08420/) (2023)
- [Improving 2D Human Pose Estimation in Rare Camera Views with Synthetic Data](https://mlsanity.com/paper/2307.06737/) (2023)
- [An Animation-based Augmentation Approach for Action Recognition from Discontinuous Video](https://mlsanity.com/paper/2404.06741/) (2024)
- [On Transferability of Driver Observation Models from Simulated to Real Environments in Autonomous Cars](https://mlsanity.com/paper/2307.16543/) (2023)
- [Dual-Branch Graph Transformer Network for 3D Human Mesh Reconstruction from Video](https://mlsanity.com/paper/2412.01179/) (2024)
- [Source Domain Subset Sampling for Semi-Supervised Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/2205.00312/) (2022)
- [GTAutoAct: An Automatic Datasets Generation Framework Based on Game Engine Redevelopment for Action Recognition](https://mlsanity.com/paper/2401.13414/) (2024)
- [VR-based generation of photorealistic synthetic data for training hand-object tracking models](https://mlsanity.com/paper/2401.17874/) (2024)
- [Posture-Driven Action Intent Inference for Playing style and Fatigue Assessment](https://mlsanity.com/paper/2507.11642/) (2025)

## Abstract (arXiv)

Recognizing Activities of Daily Living (ADL) is a vital process for intelligent assistive robots, but collecting large annotated datasets requires time-consuming temporal labeling and raises privacy concerns, e.g., if the data is collected in a real household. In this work, we explore the concept of constructing training examples for ADL recognition by playing life simulation video games and introduce the SIMS4ACTION dataset created with the popular commercial game THE SIMS 4. We build Sims4Action by specifically executing actions-of-interest in a "top-down" manner, while the gaming circumstances allow us to freely switch between environments, camera angles and subject appearances. While ADL recognition on gaming data is interesting from the theoretical perspective, the key challenge arises from transferring it to the real-world applications, such as smart-homes or assistive robotics. To meet this requirement, Sims4Action is accompanied with a GamingToReal benchmark, where the models are evaluated on real videos derived from an existing ADL dataset. We integrate two modern algorithms for video-based activity recognition in our framework, revealing the value of life simulation video games as an inexpensive and far less intrusive source of training data. However, our results also indicate that tasks involving a mixture of gaming and real data are challenging, opening a new research direction. We will make our dataset publicly available at https://github.com/aroitberg/sims4action.