# Adaptive Hyper-Graph Convolution Network for Skeleton-based Human Action Recognition with Virtual Connections

arXiv: https://arxiv.org/abs/2411.14796 · HTML: https://mlsanity.com/paper/2411.14796/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 77 references that are in the map.
- First posted on arXiv 22 Nov 2024.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [Hypergraph Convolution and Hypergraph Attention](https://mlsanity.com/paper/1901.08150/) (2019)
- [Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/1904.12659/) (2019)
- [Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition](https://mlsanity.com/paper/1904.01189/) (2019)
- [Cross-view Action Modeling, Learning and Recognition](https://mlsanity.com/paper/1405.2941/) (2014)
- [Skeleton-Based Action Recognition with Multi-Stream Adaptive Graph Convolutional Networks](https://mlsanity.com/paper/1912.06971/) (2019)
- [Constructing Stronger and Faster Baselines for Skeleton-based Action Recognition](https://mlsanity.com/paper/2106.15125/) (2021)
- [Hierarchically Decomposed Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2208.10741/) (2022)
- [Spatial Temporal Transformer Network for Skeleton-based Action Recognition](https://mlsanity.com/paper/2012.06399/) (2020)
- [Co-occurrence Feature Learning for Skeleton based Action Recognition using Regularized Deep LSTM Networks](https://mlsanity.com/paper/1603.07772/) (2016)
- [Learning Discriminative Representations for Skeleton Based Action Recognition](https://mlsanity.com/paper/2303.03729/) (2023)
- [P-CNN: Pose-based CNN Features for Action Recognition](https://mlsanity.com/paper/1506.03607/) (2015)
- [Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates](https://mlsanity.com/paper/1706.08276/) (2017)
- [SkateFormer: Skeletal-Temporal Transformer for Human Action Recognition](https://mlsanity.com/paper/2403.09508/) (2024)
- [3Mformer: Multi-order Multi-mode Transformer for Skeletal Action Recognition](https://mlsanity.com/paper/2303.14474/) (2023)
- [Differential Recurrent Neural Networks for Action Recognition](https://mlsanity.com/paper/1504.06678/) (2015)
- [Generative-based Fusion Mechanism for Multi-Modal Tracking](https://mlsanity.com/paper/2309.01728/) (2023)
- [IGFormer: Interaction Graph Transformer for Skeleton-based Human Interaction Recognition](https://mlsanity.com/paper/2207.12100/) (2022)
- [IIP-Transformer: Intra-Inter-Part Transformer for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2110.13385/) (2021)
- [Spatio-Temporal Tuples Transformer for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2201.02849/) (2022)
- [Skeletal Video Anomaly Detection using Deep Learning: Survey, Challenges and Future Directions](https://mlsanity.com/paper/2301.00114/) (2022)
- [Leveraging Spatio-Temporal Dependency for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2212.04761/) (2022)

## Built on by (most cited first, 11 of 11)

- [SkeletonAgent: An Agentic Interaction Framework for Skeleton-based Action Recognition](https://mlsanity.com/paper/2511.22433/) (2025)
- [Neurosymbolic Framework for Concept-Driven Logical Reasoning in Skeleton-Based Human Action Recognition](https://mlsanity.com/paper/2605.07140/) (2026)
- [Skeleton-Snippet Contrastive Learning with Multiscale Feature Fusion for Action Localization](https://mlsanity.com/paper/2512.16504/) (2025)
- [Skarimva: Skeleton-based Action Recognition is a Multi-view Application](https://mlsanity.com/paper/2602.23231/) (2026)
- [E2E-GNet: An End-to-End Skeleton-based Geometric Deep Neural Network for Human Motion Recognition](https://mlsanity.com/paper/2603.02477/) (2026)
- [Subspace Kernel Learning on Tensor Sequences](https://mlsanity.com/paper/2603.19546/) (2026)
- [SBF: An Effective Representation to Augment Skeleton for Video-based Human Action Recognition](https://mlsanity.com/paper/2604.03590/) (2026)
- [An Elastic Shape Variational Autoencoder for Skeleton Pose Trajectories](https://mlsanity.com/paper/2605.09231/) (2026)
- [A Neurosymbolic Framework for Interpretable Skeleton-Based Seizure Detection via Concept-Driven Logical Reasoning](https://mlsanity.com/paper/2606.21252/) (2026)
- [Partial Skeleton Visibility for Action Recognition: A Constrained Field-of-View Approach](https://mlsanity.com/paper/2607.00716/) (2026)
- [De-biasing Skeleton-based Action Recognition with Convex Hull Adaptive Shift](https://mlsanity.com/paper/2609.32454/) (2026)

## Abstract (arXiv)

The shared topology of human skeletons motivated the recent investigation of graph convolutional network (GCN) solutions for action recognition. However, most of the existing GCNs rely on the binary connection of two neighboring vertices (joints) formed by an edge (bone), overlooking the potential of constructing multi-vertex convolution structures. Although some studies have attempted to utilize hyper-graphs to represent the topology, they rely on a fixed construction strategy, which limits their adaptivity in uncovering the intricate latent relationships within the action. In this paper, we address this oversight and explore the merits of an adaptive hyper-graph convolutional network (Hyper-GCN) to achieve the aggregation of rich semantic information conveyed by skeleton vertices. In particular, our Hyper-GCN adaptively optimises the hyper-graphs during training, revealing the action-driven multi-vertex relations. Besides, virtual connections are often designed to support efficient feature aggregation, implicitly extending the spectrum of dependencies within the skeleton. By injecting virtual connections into hyper-graphs, the semantic clues of diverse action categories can be highlighted. The results of experiments conducted on the NTU-60, NTU-120, and NW-UCLA datasets demonstrate the merits of our Hyper-GCN, compared to the state-of-the-art methods. The code is available at https://github.com/6UOOON9/Hyper-GCN.