# Cross-view Multimodal Vision-Based Assessment Framework for Traditional Chinese Medicine Rehabilitation Training

arXiv: https://arxiv.org/abs/2606.28104 · HTML: https://mlsanity.com/paper/2606.28104/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 65 references that are in the map.
- First posted on arXiv 26 Jun 2026.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [MediaPipe Hands: On-device Real-time Hand Tracking](https://mlsanity.com/paper/2006.10214/) (2020)
- [FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessment](https://mlsanity.com/paper/2204.03646/) (2022)
- [What and How Well You Performed? A Multitask Learning Approach to Action Quality Assessment](https://mlsanity.com/paper/1904.04346/) (2019)
- [UmeTrack: Unified multi-view end-to-end hand tracking for VR](https://mlsanity.com/paper/2211.00099/) (2022)
- [Hierarchical Temporal Transformer for 3D Hand Pose Estimation and Action Recognition from Egocentric RGB Videos](https://mlsanity.com/paper/2209.09484/) (2022)
- [LOGO: A Long-Form Video Dataset for Group Action Quality Assessment](https://mlsanity.com/paper/2404.05029/) (2024)
- [A Survey on Video Action Recognition in Sports: Datasets, Methods and Applications](https://mlsanity.com/paper/2206.01038/) (2022)
- [A Deep Learning Framework for Assessing Physical Rehabilitation Exercises](https://mlsanity.com/paper/1901.10435/) (2019)
- [FineParser: A Fine-grained Spatio-temporal Action Parser for Human-centric Action Quality Assessment](https://mlsanity.com/paper/2405.06887/) (2024)
- [A Comprehensive Survey of Action Quality Assessment: Method and Benchmark](https://mlsanity.com/paper/2412.11149/) (2024)
- [Geometric Features Informed Multi-person Human-object Interaction Recognition in Videos](https://mlsanity.com/paper/2207.09425/) (2022)
- [A Survey of Video-based Action Quality Assessment](https://mlsanity.com/paper/2204.09271/) (2022)
- [PHI: Bridging Domain Shift in Long-Term Action Quality Assessment via Progressive Hierarchical Instruction](https://mlsanity.com/paper/2505.19972/) (2025)
- [Learning Sparse Temporal Video Mapping for Action Quality Assessment in Floor Gymnastics](https://mlsanity.com/paper/2301.06103/) (2023)
- [3D Human Pose Estimation in Multi-View Operating Room Videos Using Differentiable Camera Projections](https://mlsanity.com/paper/2210.11826/) (2022)
- [A Medical Low-Back Pain Physical Rehabilitation Dataset for Human Body Movement Analysis](https://mlsanity.com/paper/2407.00521/) (2024)
- [Geometric Features Enhanced Human-Object Interaction Detection](https://mlsanity.com/paper/2406.18691/) (2024)
- [Adaptive Graph Learning from Spatial Information for Surgical Workflow Anticipation](https://mlsanity.com/paper/2412.06454/) (2024)
- [TaiChi Action Capture and Performance Analysis with Multi-view RGB Cameras](https://mlsanity.com/paper/2306.14490/) (2023)

## Abstract (arXiv)

Vision-based assessment can provide convenient and cost-effective evaluation in Traditional Chinese Medicine (TCM) rehabilitation training, where action quality assessment (AQA) from computer vision offers a promising solution. Existing automatic AQA frameworks for physical therapy typically rely on skeletal data captured from a single viewpoint, which is inefficient for TCM techniques such as acupuncture or Tuina that involve dense hand self-occlusion and complex hand-object interactions. To address these challenges, we propose CME-AQA, a cross-view, multimodal vision-based assessment framework that integrates visual-pose fusion to enhance understanding of environmental context and leverages both first-person and third-person videos during training to improve inference robustness. We collected two dual-view datasets, TCM-AQA61-A (Acupuncture) and TCM-AQA61-T (Tuina), each containing synchronized first-person and third-person recordings of 61 subjects with expert annotations. Experimental results show that our approach achieves superior or comparable mean performance against competitive baselines, achieving over 10% relative improvement in weighted F1 over the best competing method on key rating tasks such as Needle Depth and Quick Needle Insertion, while also reducing mean absolute error in quantitative measures such as insertion time and manipulation frequency. Testing on a CPR dataset further demonstrates comparable performance on several posture-based criteria, suggesting applicability to related structured simulated clinical skill assessments where participant motion is central to evaluation. Overall, CME-AQA enhances assessment accuracy for structured TCM rehabilitation training and facilitates more convenient and effective training-oriented skill evaluation.