# VSMP-IMU: Video-Grounded Semantic Motion Programs for Sensor-Aware Synthetic IMU Generation

arXiv: https://arxiv.org/abs/2608.05782 · HTML: https://mlsanity.com/paper/2608.05782/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 37 references that are in the map.
- First posted on arXiv 6 Aug 2026.

## Builds on (most cited first, 15 of 15)

- [FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding](https://mlsanity.com/paper/2004.06704/) (2020)
- [Data Augmentation of Wearable Sensor Data for Parkinson's Disease Monitoring using Convolutional Neural Networks](https://mlsanity.com/paper/1706.00527/) (2017)
- [Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordance](https://mlsanity.com/paper/2403.18036/) (2024)
- [Assessing the State of Self-Supervised Human Activity Recognition using Wearables](https://mlsanity.com/paper/2202.12938/) (2022)
- [Contrastive Predictive Coding for Human Activity Recognition](https://mlsanity.com/paper/2012.05333/) (2020)
- [HY-Motion 1.0: Scaling Flow Matching Models for Text-To-Motion Generation](https://mlsanity.com/paper/2512.23464/) (2025)
- [Semantic-Discriminative Mixup for Generalizable Sensor-based Cross-domain Activity Recognition](https://mlsanity.com/paper/2206.06629/) (2022)
- [Motion Question Answering via Modular Motion Programs](https://mlsanity.com/paper/2305.08953/) (2023)
- [Past, Present, and Future of Sensor-Based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task](https://mlsanity.com/paper/2411.14452/) (2024)
- [Hierarchical Motion Understanding via Motion Programs](https://mlsanity.com/paper/2104.11216/) (2021)
- [Limitations in Employing Natural Language Supervision for Sensor-Based Human Activity Recognition -- And Ways to Overcome Them](https://mlsanity.com/paper/2408.12023/) (2024)
- [Therbligs in Action: Video Understanding through Motion Primitives](https://mlsanity.com/paper/2304.03631/) (2023)
- [LEAP: LLM-Generation of Egocentric Action Programs](https://mlsanity.com/paper/2312.00055/) (2023)
- [Scaling Human Activity Recognition: A Comparative Evaluation of Synthetic Data Generation and Augmentation Techniques](https://mlsanity.com/paper/2506.07612/) (2025)
- [Video2IMU: Realistic IMU features and signals from videos](https://mlsanity.com/paper/2202.06547/) (2022)

## Abstract (arXiv)

Wearable human activity recognition (HAR) is often limited by the scarcity of labeled sensor data, especially in low-resource, class-imbalanced, and subject-generalization settings. Synthetic IMU generation can reduce this dependency and enhance HAR machine learning model's performance, but existing approaches face a trade-off without addressing all factors: video-driven methods are visually grounded but sensitive to pose-estimation errors, while text-driven methods are controllable but often weakly grounded in how activities are actually performed. We present VSMP-IMU, a video-grounded framework for controllable synthetic IMU generation based on a structured Semantic Motion Program (SMP), which separates activity-defining semantics from label-preserving variation. Given an input video, VSMP-IMU extracts and augments an SMP, uses it to synthesize motion, converts the motion into virtual IMU signals, and grounds the resulting signals to the target wearable domain. We evaluate VSMP-IMU against state-of-the-art synthetic data generation methods on five public IMU-HAR datasets under leave-one-person-out evaluation. VSMP-IMU achieves an average Macro-F1 of 78.33%, improving over real-only training by 9.77% and over the strongest prior synthetic baseline by 4.04%. In low-resource settings with reduced training data-samples, it improves over real-only training by 18.54% and over the strongest prior synthetic baselines by more than 6% on average. Under long-tail evaluation in imbalanced datasets, it improves tail-class Macro-F1 by 19.86% over Real-only training and by 4.76% over SOTA. These results show that structured video-grounded semantics provide a practical foundation for controllable, wearable-relevant synthetic sensor data generation.