# Generating Virtual On-body Accelerometer Data from Virtual Textual Descriptions for Human Activity Recognition

arXiv: https://arxiv.org/abs/2305.03187 · HTML: https://mlsanity.com/paper/2305.03187/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 42 references that are in the map.
- First posted on arXiv 4 May 2023.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [Executing your Commands via Motion Diffusion in Latent Space](https://mlsanity.com/paper/2212.04048/) (2022)
- [Decoupling Human and Camera Motion from Videos in the Wild](https://mlsanity.com/paper/2302.12827/) (2023)
- [Unpaired Motion Style Transfer from Video to Animation](https://mlsanity.com/paper/2005.05751/) (2020)
- [IMUTube: Automatic Extraction of Virtual on-body Accelerometry from Video for Human Activity Recognition](https://mlsanity.com/paper/2006.05675/) (2020)

## Built on by (most cited first, 19 of 19)

- [Large Language Models for Time Series: A Survey](https://mlsanity.com/paper/2402.01801/) (2024)
- [UniMTS: Unified Pre-training for Motion Time Series](https://mlsanity.com/paper/2410.19818/) (2024)
- [IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition](https://mlsanity.com/paper/2402.01049/) (2024)
- [Past, Present, and Future of Sensor-Based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task](https://mlsanity.com/paper/2411.14452/) (2024)
- [A Survey on Multimodal Wearable Sensor-based Human Action Recognition](https://mlsanity.com/paper/2404.15349/) (2024)
- [Using Large Language Models to Compare Explainable Models for Smart Home Human Activity Recognition](https://mlsanity.com/paper/2408.06352/) (2024)
- [Enhancing Inertial Hand based HAR through Joint Representation of Language, Pose and Synthetic IMUs](https://mlsanity.com/paper/2406.01316/) (2024)
- [Towards Generalizable Human Activity Recognition: A Survey](https://mlsanity.com/paper/2508.12213/) (2025)
- [Physical Self-Supervised Learning: IMU Sensing without Manual Labels](https://mlsanity.com/paper/2607.18361/) (2026)
- [Boosting Automatic Exercise Evaluation Through Musculoskeletal Simulation-Based IMU Data Augmentation](https://mlsanity.com/paper/2505.24415/) (2025)
- [Physically Plausible Data Augmentations for Wearable IMU-based Human Activity Recognition Using Physics Simulation](https://mlsanity.com/paper/2508.13284/) (2025)
- [AnyMo: Geometry-Aware Setup-Agnostic Modeling of Human Motion in the Wild](https://mlsanity.com/paper/2605.22715/) (2026)
- [ContextGPT: Infusing LLMs Knowledge into Neuro-Symbolic Activity Recognition Models](https://mlsanity.com/paper/2403.06586/) (2024)
- [AI-Generated Fall Data: Assessing LLMs and Diffusion Model for Wearable Fall Detection](https://mlsanity.com/paper/2505.04660/) (2025)
- [Challenges and Limitations of Generative AI in Synthesizing Wearable Sensor Data](https://mlsanity.com/paper/2505.14206/) (2025)
- [XTransfer: Modality-Agnostic Few-Shot Model Transfer for Human Sensing at the Edge](https://mlsanity.com/paper/2506.22726/) (2025)
- [Refining Diffusion Models for Motion Synthesis with an Acceleration Loss to Generate Realistic IMU Data](https://mlsanity.com/paper/2512.08859/) (2025)
- [Coverage-Aware Virtual IMU Augmentation for Low-Resource Human Activity Recognition](https://mlsanity.com/paper/2609.16768/) (2026)
- [CHEAR: Confusion-Hierarchy Evidence-Aware Reasoning for Cost-Efficient Human Activity Recognition](https://mlsanity.com/paper/2610.15298/) (2026)

## Abstract (arXiv)

The development of robust, generalized models in human activity recognition (HAR) has been hindered by the scarcity of large-scale, labeled data sets. Recent work has shown that virtual IMU data extracted from videos using computer vision techniques can lead to substantial performance improvements when training HAR models combined with small portions of real IMU data. Inspired by recent advances in motion synthesis from textual descriptions and connecting Large Language Models (LLMs) to various AI models, we introduce an automated pipeline that first uses ChatGPT to generate diverse textual descriptions of activities. These textual descriptions are then used to generate 3D human motion sequences via a motion synthesis model, T2M-GPT, and later converted to streams of virtual IMU data. We benchmarked our approach on three HAR datasets (RealWorld, PAMAP2, and USC-HAD) and demonstrate that the use of virtual IMU training data generated using our new approach leads to significantly improved HAR model performance compared to only using real IMU data. Our approach contributes to the growing field of cross-modality transfer methods and illustrate how HAR models can be improved through the generation of virtual training data that do not require any manual effort.