# HieroAction: Hierarchically Guided VLM for Fine-Grained Action Analysis

arXiv: https://arxiv.org/abs/2508.16942 · HTML: https://mlsanity.com/paper/2508.16942/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 23 Aug 2025.

## Builds on (most cited first, 22 of 22)

- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Multimodal Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2302.00923/) (2023)
- [VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding](https://mlsanity.com/paper/2501.13106/) (2025)
- [Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning](https://mlsanity.com/paper/2403.16999/) (2024)
- [NVILA: Efficient Frontier Visual Language Models](https://mlsanity.com/paper/2412.04468/) (2024)
- [ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Model](https://mlsanity.com/paper/2502.14420/) (2025)
- [FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessment](https://mlsanity.com/paper/2204.03646/) (2022)
- [What and How Well You Performed? A Multitask Learning Approach to Action Quality Assessment](https://mlsanity.com/paper/1904.04346/) (2019)
- [MotionLLM: Understanding Human Behaviors from Human Motions and Videos](https://mlsanity.com/paper/2405.20340/) (2024)
- [Temporal Action Segmentation: An Analysis of Modern Techniques](https://mlsanity.com/paper/2210.10352/) (2022)
- [Uncertainty-aware Score Distribution Learning for Action Quality Assessment](https://mlsanity.com/paper/2006.07665/) (2020)
- [Group-aware Contrastive Regression for Action Quality Assessment](https://mlsanity.com/paper/2108.07797/) (2021)
- [LOGO: A Long-Form Video Dataset for Group Action Quality Assessment](https://mlsanity.com/paper/2404.05029/) (2024)
- [A Comprehensive Survey of Action Quality Assessment: Method and Benchmark](https://mlsanity.com/paper/2412.11149/) (2024)
- [CoFInAl: Enhancing Action Quality Assessment with Coarse-to-Fine Instruction Alignment](https://mlsanity.com/paper/2404.13999/) (2024)
- [VerIPO: Cultivating Long Reasoning in Video-LLMs via Verifier-Gudied Iterative Policy Optimization](https://mlsanity.com/paper/2505.19000/) (2025)
- [DIP-R1: Deep Inspection and Perception with RL Looking Through and Understanding Complex Scenes](https://mlsanity.com/paper/2505.23179/) (2025)
- [LLAVIDAL: A Large LAnguage VIsion Model for Daily Activities of Living](https://mlsanity.com/paper/2406.09390/) (2024)
- [Domain Adaptation of VLM for Soccer Video Understanding](https://mlsanity.com/paper/2505.13860/) (2025)
- [PATS: Proficiency-Aware Temporal Sampling for Multi-View Sports Skill Assessment](https://mlsanity.com/paper/2506.04996/) (2025)
- [MS-LSTM: Exploring Spatiotemporal Multiscale Representations in Video Prediction Domain](https://mlsanity.com/paper/2304.07724/) (2023)
- [FineCausal: A Causal-Based Framework for Interpretable Fine-Grained Action Quality Assessment](https://mlsanity.com/paper/2503.23911/) (2025)

## Built on by (most cited first, 1 of 1)

- [Can Vision Language Models Judge Action Quality? An Empirical Evaluation](https://mlsanity.com/paper/2604.08294/) (2026)

## Abstract (arXiv)

Evaluating human actions with clear and detailed feedback is important in areas such as sports, healthcare, and robotics, where decisions rely not only on final outcomes but also on interpretable reasoning. However, most existing methods provide only a final score without explanation or detailed analysis, limiting their practical applicability. To address this, we introduce HieroAction, a vision-language model that delivers accurate and structured assessments of human actions. HieroAction builds on two key ideas: (1) Stepwise Action Reasoning, a tailored chain of thought process designed specifically for action assessment, which guides the model to evaluate actions step by step, from overall recognition through sub action analysis to final scoring, thus enhancing interpretability and structured understanding; and (2) Hierarchical Policy Learning, a reinforcement learning strategy that enables the model to learn fine grained sub action dynamics and align them with high level action quality, thereby improving scoring precision. The reasoning pathway structures the evaluation process, while policy learning refines each stage through reward based optimization. Their integration ensures accurate and interpretable assessments, as demonstrated by superior performance across multiple benchmark datasets. Code will be released upon acceptance.