# Cross-Modal Video to Body-joints Augmentation for Rehabilitation Exercise Quality Assessment

arXiv: https://arxiv.org/abs/2306.09546 · HTML: https://mlsanity.com/paper/2306.09546/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 25 references that are in the map.
- First posted on arXiv 15 Jun 2023.

## Builds on (most cited first, 9 of 9)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [3D human pose estimation in video with temporal convolutions and semi-supervised training](https://mlsanity.com/paper/1811.11742/) (2018)
- [A Deep Learning Framework for Assessing Physical Rehabilitation Exercises](https://mlsanity.com/paper/1901.10435/) (2019)
- [A Review of Computational Approaches for Evaluation of Rehabilitation Exercises](https://mlsanity.com/paper/2003.08767/) (2020)
- [Paired Cross-Modal Data Augmentation for Fine-Grained Image-to-Text Retrieval](https://mlsanity.com/paper/2207.14428/) (2022)
- [Rehabilitation Exercise Repetition Segmentation and Counting using Skeletal Body Joints](https://mlsanity.com/paper/2304.09735/) (2023)
- [Parkinson's Disease Assessment from a Wrist-Worn Wearable Sensor in Free-Living Conditions: Deep Ensemble Learning and Visualization](https://mlsanity.com/paper/1808.02870/) (2018)

## Built on by (most cited first, 3 of 3)

- [Engagement Measurement Based on Facial Landmarks and Spatial-Temporal Graph Convolutional Networks](https://mlsanity.com/paper/2403.17175/) (2024)
- [Rehabilitation Exercise Quality Assessment and Feedback Generation Using Large Language Models with Prompt Engineering](https://mlsanity.com/paper/2505.18412/) (2025)
- [Rehabilitation Exercise Quality Assessment through Supervised Contrastive Learning with Hard and Soft Negatives](https://mlsanity.com/paper/2403.02772/) (2024)

## Abstract (arXiv)

Exercise-based rehabilitation programs have been shown to enhance quality of life and reduce mortality and rehospitalizations. AI-driven virtual rehabilitation programs enable patients to complete exercises independently at home while AI algorithms can analyze exercise data to provide feedback to patients and report their progress to clinicians. This paper introduces a novel approach to assessing the quality of rehabilitation exercises using RGB video. Sequences of skeletal body joints are extracted from consecutive RGB video frames and analyzed by many-to-one sequential neural networks to evaluate exercise quality. Existing datasets for exercise rehabilitation lack adequate samples for training deep sequential neural networks to generalize effectively. A cross-modal data augmentation approach is proposed to resolve this problem. Visual augmentation techniques are applied to video data, and body joints extracted from the resulting augmented videos are used for training sequential neural networks. Extensive experiments conducted on the KInematic assessment of MOvement and clinical scores for remote monitoring of physical REhabilitation (KIMORE) dataset, demonstrate the superiority of the proposed method over previous baseline approaches. The ablation study highlights a significant enhancement in exercise quality assessment following cross-modal augmentation.