# Autoregressive Adaptive Hypergraph Transformer for Skeleton-based Activity Recognition

arXiv: https://arxiv.org/abs/2411.05692 · HTML: https://mlsanity.com/paper/2411.05692/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 50 references that are in the map.
- First posted on arXiv 8 Nov 2024.

## Builds on (most cited first, 30 of 34)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Activating More Pixels in Image Super-Resolution Transformer](https://mlsanity.com/paper/2205.04437/) (2022)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [Temporal Action Detection with Structured Segment Networks](https://mlsanity.com/paper/1704.06228/) (2017)
- [Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition](https://mlsanity.com/paper/1607.07043/) (2016)
- [Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition](https://mlsanity.com/paper/1904.01189/) (2019)
- [Cross-view Action Modeling, Learning and Recognition](https://mlsanity.com/paper/1405.2941/) (2014)
- [A Spatio-temporal Transformer for 3D Human Motion Prediction](https://mlsanity.com/paper/2004.08692/) (2020)
- [A New Representation of Skeleton Sequences for 3D Action Recognition](https://mlsanity.com/paper/1703.03492/) (2017)
- [Interpretable 3D Human Action Analysis with Temporal Convolutional Networks](https://mlsanity.com/paper/1704.04516/) (2017)
- [Constructing Stronger and Faster Baselines for Skeleton-based Action Recognition](https://mlsanity.com/paper/2106.15125/) (2021)
- [Hierarchically Decomposed Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2208.10741/) (2022)
- [Spatial Temporal Transformer Network for Skeleton-based Action Recognition](https://mlsanity.com/paper/2012.06399/) (2020)
- [Co-occurrence Feature Learning for Skeleton based Action Recognition using Regularized Deep LSTM Networks](https://mlsanity.com/paper/1603.07772/) (2016)
- [Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors](https://mlsanity.com/paper/1505.04868/) (2015)
- [Skeleton-based Action Recognition with Convolutional Neural Networks](https://mlsanity.com/paper/1704.07595/) (2017)
- [Masked Motion Predictors are Strong 3D Action Representation Learners](https://mlsanity.com/paper/2308.07092/) (2023)
- [3Mformer: Multi-order Multi-mode Transformer for Skeletal Action Recognition](https://mlsanity.com/paper/2303.14474/) (2023)
- [Hypergraph Transformer for Skeleton-based Action Recognition](https://mlsanity.com/paper/2211.09590/) (2022)
- [IGFormer: Interaction Graph Transformer for Skeleton-based Human Interaction Recognition](https://mlsanity.com/paper/2207.12100/) (2022)
- [A Comparative Review of Recent Kinect-based Action Recognition Algorithms](https://mlsanity.com/paper/1906.09955/) (2019)
- [SkeletonMAE: Spatial-Temporal Masked Autoencoders for Self-supervised Skeleton Action Recognition](https://mlsanity.com/paper/2209.02399/) (2022)
- [Focal and Global Spatial-Temporal Transformer for Skeleton-based Action Recognition](https://mlsanity.com/paper/2210.02693/) (2022)
- [IIP-Transformer: Intra-Inter-Part Transformer for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2110.13385/) (2021)
- [Self-Attention Network for Skeleton-based Human Action Recognition](https://mlsanity.com/paper/1912.08435/) (2019)

## Built on by (most cited first, 4 of 4)

- [Skeleton-Snippet Contrastive Learning with Multiscale Feature Fusion for Action Localization](https://mlsanity.com/paper/2512.16504/) (2025)
- [Beyond Pairwise Relations: Dynamic Manipulation Hypergraphs for Vision-Based Human Activity Recognition](https://mlsanity.com/paper/2607.14350/) (2026)
- [Structure over Depth: A Single-Block Spatio-Temporal Transformer for Multi-Entity Reasoning](https://mlsanity.com/paper/2607.23077/) (2026)
- [FLASH: Efficient Impact Fall Detection with Unified Hypergraph State-Space Model](https://mlsanity.com/paper/2607.25791/) (2026)

## Abstract (arXiv)

Extracting multiscale contextual information and higher-order correlations among skeleton sequences using Graph Convolutional Networks (GCNs) alone is inadequate for effective action classification. Hypergraph convolution addresses the above issues but cannot harness the long-range dependencies. The transformer proves to be effective in capturing these dependencies and making complex contextual features accessible. We propose an Autoregressive Adaptive HyperGraph Transformer (AutoregAd-HGformer) model for in-phase (autoregressive and discrete) and out-phase (adaptive) hypergraph generation. The vector quantized in-phase hypergraph equipped with powerful autoregressive learned priors produces a more robust and informative representation suitable for hyperedge formation. The out-phase hypergraph generator provides a model-agnostic hyperedge learning technique to align the attributes with input skeleton embedding. The hybrid (supervised and unsupervised) learning in AutoregAd-HGformer explores the action-dependent feature along spatial, temporal, and channel dimensions. The extensive experimental results and ablation study indicate the superiority of our model over state-of-the-art hypergraph architectures on the NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets.