# Fine-Grained Action Recognition with Cross-Attentive Latent Sparse Experts

arXiv: https://arxiv.org/abs/2608.13458 · HTML: https://mlsanity.com/paper/2608.13458/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 13 Aug 2026.

## Builds on (most cited first, 29 of 29)

- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](https://mlsanity.com/paper/2112.01526/) (2021)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [Temporal Relational Reasoning in Videos](https://mlsanity.com/paper/1711.08496/) (2017)
- [Expanding Language-Image Pretrained Models for General Video Recognition](https://mlsanity.com/paper/2208.02816/) (2022)
- [ActionCLIP: A New Paradigm for Video Action Recognition](https://mlsanity.com/paper/2109.08472/) (2021)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [Revisiting Skeleton-based Action Recognition](https://mlsanity.com/paper/2104.13586/) (2021)
- [FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding](https://mlsanity.com/paper/2004.06704/) (2020)
- [UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning](https://mlsanity.com/paper/2201.04676/) (2022)
- [Temporal Pyramid Network for Action Recognition](https://mlsanity.com/paper/2004.03548/) (2020)
- [AIM: Adapting Image Models for Efficient Video Action Recognition](https://mlsanity.com/paper/2302.03024/) (2023)
- [Large-scale weakly-supervised pre-training for video action recognition](https://mlsanity.com/paper/1905.00561/) (2019)
- [PYSKL: Towards Good Practices for Skeleton Action Recognition](https://mlsanity.com/paper/2205.09443/) (2022)
- [D-FINE: Redefine Regression Task in DETRs as Fine-grained Distribution Refinement](https://mlsanity.com/paper/2410.13842/) (2024)
- [Temporal Query Networks for Fine-grained Video Understanding](https://mlsanity.com/paper/2104.09496/) (2021)
- [Use Your Head: Improving Long-Tail Video Recognition](https://mlsanity.com/paper/2304.01143/) (2023)
- [TAG-Head: Time-Aligned Graph Head for Plug-and-Play Fine-grained Action Recognition](https://mlsanity.com/paper/2604.11498/) (2026)

## Abstract (arXiv)

Fine-grained human action recognition (FHAR) must distinguish visually similar actions that differ mainly in body configuration, timing, or local appearance. RGB representations retain visual context but often suppress joint-level geometry, whereas skeleton representations encode kinematics but discard dense spatial detail. We introduce FineX, which factorizes fine-grained cues into RGB appearance, pose heatmap geometry, and skeletal-graph topology. Pairwise cross-attention enables symmetric, stream-preserving information exchange, followed by a streamwise latent sparse Mixture-of-Experts that routes each representation to a content-dependent subset of shared experts, regularized by a load-balancing objective. FineX achieves state-of-the-art results on Gym99, Gym288, and Diving48. On the long-tailed Gym288, it raises mean class accuracy from 68.6% to 76.2% (+7.6 points) without textual supervision or large-scale vision-language pre-training, demonstrating the benefit of structured visual-pose-graph fusion and conditional expert refinement for FHAR.