# Two-Stage Multi-Modal Fusion with Adaptive Alignment for Action Quality Assessment

arXiv: https://arxiv.org/abs/2607.07438 · HTML: https://mlsanity.com/paper/2607.07438/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 72 references that are in the map.
- First posted on arXiv 8 Jul 2026.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021)
- [LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment](https://mlsanity.com/paper/2310.01852/) (2023)
- [FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessment](https://mlsanity.com/paper/2204.03646/) (2022)
- [VideoPrism: A Foundational Visual Encoder for Video Understanding](https://mlsanity.com/paper/2402.13217/) (2024)
- [Neural Collapse: A Review on Modelling Principles and Generalization](https://mlsanity.com/paper/2206.04041/) (2022)
- [Learning To Score Olympic Events](https://mlsanity.com/paper/1611.05125/) (2016)
- [Group-aware Contrastive Regression for Action Quality Assessment](https://mlsanity.com/paper/2108.07797/) (2021)
- [Multimodal Alignment and Fusion: A Survey](https://mlsanity.com/paper/2411.17040/) (2024)
- [TSA-Net: Tube Self-Attention Network for Action Quality Assessment](https://mlsanity.com/paper/2201.03746/) (2022)
- [Seed-X: Building Strong Multilingual Translation LLM with 7B Parameters](https://mlsanity.com/paper/2507.13618/) (2025)
- [Skating-Mixer: Long-Term Sport Audio-Visual Modeling with MLPs](https://mlsanity.com/paper/2203.03990/) (2022)
- [Multimodal Action Quality Assessment](https://mlsanity.com/paper/2402.09444/) (2024)
- [A Comprehensive Survey of Action Quality Assessment: Method and Benchmark](https://mlsanity.com/paper/2412.11149/) (2024)
- [Auto-Encoding Score Distribution Regression for Action Quality Assessment](https://mlsanity.com/paper/2111.11029/) (2021)
- [Accommodating Audio Modality in CLIP for Multimodal Processing](https://mlsanity.com/paper/2303.06591/) (2023)
- [Piano Skills Assessment](https://mlsanity.com/paper/2101.04884/) (2021)
- [MAGR: Manifold-Aligned Graph Regularization for Continual Action Quality Assessment](https://mlsanity.com/paper/2403.04398/) (2024)
- [RICA2: Rubric-Informed, Calibrated Assessment of Actions](https://mlsanity.com/paper/2408.02138/) (2024)
- [Continual Action Assessment via Task-Consistent Score-Discriminative Feature Distribution Modeling](https://mlsanity.com/paper/2309.17105/) (2023)
- [PHI: Bridging Domain Shift in Long-Term Action Quality Assessment via Progressive Hierarchical Instruction](https://mlsanity.com/paper/2505.19972/) (2025)
- [Interpretable Long-term Action Quality Assessment](https://mlsanity.com/paper/2408.11687/) (2024)
- [A Decade of Action Quality Assessment: Largest Systematic Survey of Trends, Challenges, and Future Directions](https://mlsanity.com/paper/2502.02817/) (2025)
- [FineCausal: A Causal-Based Framework for Interpretable Fine-Grained Action Quality Assessment](https://mlsanity.com/paper/2503.23911/) (2025)
- [BriMA: Bridged Modality Adaptation for Multi-Modal Continual Action Quality Assessment](https://mlsanity.com/paper/2602.19170/) (2026)

## Abstract (arXiv)

Action Quality Assessment (AQA) aims to evaluate how well a person performs a movement, which is essential in applications such as sports scoring, skill assessment, and healthcare. However, unimodal approaches often struggle to capture subtle cues of movement quality in real-world settings. Although multi-modal inputs provide complementary information, existing methods still face two major challenges: heterogeneous modalities often lead to cross-modal misalignment and unstable fusion, and reliable multi-modal annotation is costly, resulting in limited dataset diversity. To address these challenges, we propose DualAlign, a two-stage multi-modal fusion framework with adaptive alignment. The framework first constructs a coherent visual representation by maximizing shared structural information across RGB video, optical flow, and skeleton modalities. Textual semantics are then incorporated after visual stabilization, allowing high-level descriptions to complement rather than distort the underlying visual manifold. To evaluate the framework under realistic multi-modal conditions, we introduce MM--JDM, a movement-quality assessment dataset integrating RGB videos, optical flow, skeleton sequences, and structured text. MM--JDM naturally exhibits modality noise, class imbalance, and label scarcity, making it a challenging benchmark for studying multi-modal fusion and alignment. Extensive experiments show that DualAlign improves average correlation on MM--JDM by 21.16% over the state-of-the-art methods and achieves gains of 3.53% and 5.95% on the RG and Fis-V benchmarks, respectively. DualAlign also remains robust under missing-modality and label-scarce conditions.