# IMUTube: Automatic Extraction of Virtual on-body Accelerometry from Video for Human Activity Recognition

arXiv: https://arxiv.org/abs/2006.05675 · HTML: https://mlsanity.com/paper/2006.05675/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2020.

## Built on by (most cited first, 22 of 22)

- [Assessing the State of Self-Supervised Human Activity Recognition using Wearables](https://mlsanity.com/paper/2202.12938/) (2022)
- [Improving Deep Learning for HAR with shallow LSTMs](https://mlsanity.com/paper/2108.00702/) (2021)
- [IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition](https://mlsanity.com/paper/2402.01049/) (2024)
- [Generating Virtual On-body Accelerometer Data from Virtual Textual Descriptions for Human Activity Recognition](https://mlsanity.com/paper/2305.03187/) (2023)
- [Past, Present, and Future of Sensor-Based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task](https://mlsanity.com/paper/2411.14452/) (2024)
- [Cross-Domain HAR: Few Shot Transfer Learning for Human Activity Recognition](https://mlsanity.com/paper/2310.14390/) (2023)
- [COMODO: Cross-Modal Video-to-IMU Distillation for Efficient Egocentric Human Activity Recognition](https://mlsanity.com/paper/2503.07259/) (2025)
- [Fast and Robust Video-Based Exercise Classification via Body Pose Tracking and Scalable Multivariate Time Series Classifiers](https://mlsanity.com/paper/2210.00507/) (2022)
- [Investigating Enhancements to Contrastive Predictive Coding for Human Activity Recognition](https://mlsanity.com/paper/2211.06173/) (2022)
- [Enhancing Inertial Hand based HAR through Joint Representation of Language, Pose and Synthetic IMUs](https://mlsanity.com/paper/2406.01316/) (2024)
- [ConvBoost: Boosting ConvNets for Sensor-based Activity Recognition](https://mlsanity.com/paper/2305.13541/) (2023)
- [IMUCoCo: Enabling Flexible On-Body IMU Placement for Human Pose Estimation and Activity Recognition](https://mlsanity.com/paper/2508.01894/) (2025)
- [Fine-grained Human Activity Recognition Using Virtual On-body Acceleration Data](https://mlsanity.com/paper/2211.01342/) (2022)
- [Don't freeze: Finetune encoders for better Self-Supervised HAR](https://mlsanity.com/paper/2307.01168/) (2023)
- [On the Benefit of Generative Foundation Models for Human Activity Recognition](https://mlsanity.com/paper/2310.12085/) (2023)
- [SudokuSens: Enhancing Deep Learning Robustness for IoT Sensing Applications using a Generative Approach](https://mlsanity.com/paper/2402.02275/) (2024)
- [Physical Self-Supervised Learning: IMU Sensing without Manual Labels](https://mlsanity.com/paper/2607.18361/) (2026)
- [PhysiQ: Off-site Quality Assessment of Exercise in Physical Therapy](https://mlsanity.com/paper/2211.08245/) (2022)
- [Interpersonal Distance Tracking with mmWave Radar and IMUs](https://mlsanity.com/paper/2303.12798/) (2023)
- [TxP: Reciprocal Generation of Ground Pressure Dynamics and Activity Descriptions for Improving Human Activity Recognition](https://mlsanity.com/paper/2505.02052/) (2025)
- [Matching Skeleton-based Activity Representations with Heterogeneous Signals for HAR](https://mlsanity.com/paper/2503.14547/) (2025)
- [SImpHAR: Advancing impedance-based human activity recognition using 3D simulation and text-to-motion models](https://mlsanity.com/paper/2507.06405/) (2025)

## Abstract (arXiv)

The lack of large-scale, labeled data sets impedes progress in developing robust and generalized predictive models for on-body sensor-based human activity recognition (HAR). Labeled data in human activity recognition is scarce and hard to come by, as sensor data collection is expensive, and the annotation is time-consuming and error-prone. To address this problem, we introduce IMUTube, an automated processing pipeline that integrates existing computer vision and signal processing techniques to convert videos of human activity into virtual streams of IMU data. These virtual IMU streams represent accelerometry at a wide variety of locations on the human body. We show how the virtually-generated IMU data improves the performance of a variety of models on known HAR datasets. Our initial results are very promising, but the greater promise of this work lies in a collective approach by the computer vision, signal processing, and activity recognition communities to extend this work in ways that we outline. This should lead to on-body, sensor-based HAR becoming yet another success story in large-dataset breakthroughs in recognition.