# Learning Mutual Excitation for Hand-to-Hand and Human-to-Human Interaction Recognition

arXiv: https://arxiv.org/abs/2402.02431 · HTML: https://mlsanity.com/paper/2402.02431/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 60 references that are in the map.
- First posted on arXiv 4 Feb 2024.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/1904.12659/) (2019)
- [Human Action Recognition and Prediction: A Survey](https://mlsanity.com/paper/1806.11230/) (2018)
- [Human Action Recognition from Various Data Modalities: A Review](https://mlsanity.com/paper/2012.11866/) (2020)
- [Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition](https://mlsanity.com/paper/1607.07043/) (2016)
- [Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition](https://mlsanity.com/paper/1904.01189/) (2019)
- [A New Representation of Skeleton Sequences for 3D Action Recognition](https://mlsanity.com/paper/1703.03492/) (2017)
- [Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation](https://mlsanity.com/paper/1804.06055/) (2018)
- [Skeleton-based Action Recognition via Spatial and Temporal Transformer Networks](https://mlsanity.com/paper/2008.07404/) (2020)
- [Intel RealSense Stereoscopic Depth Cameras](https://mlsanity.com/paper/1705.05548/) (2017)
- [Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition](https://mlsanity.com/paper/2007.14690/) (2020)
- [Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching](https://mlsanity.com/paper/1911.04131/) (2019)
- [Skeleton-Based Human Action Recognition with Global Context-Aware Attention LSTM Networks](https://mlsanity.com/paper/1707.05740/) (2017)
- [IGFormer: Interaction Graph Transformer for Skeleton-based Human Interaction Recognition](https://mlsanity.com/paper/2207.12100/) (2022)
- [Consistent 3D Hand Reconstruction in Video via self-supervised Learning](https://mlsanity.com/paper/2201.09548/) (2022)
- [Interaction Relational Network for Mutual Action Recognition](https://mlsanity.com/paper/1910.04963/) (2019)
- [Spatio-Temporal Inception Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2011.13322/) (2020)
- [RGB-D-based Action Recognition Datasets: A Survey](https://mlsanity.com/paper/1601.05511/) (2016)
- [Dynamic Dense Graph Convolutional Network for Skeleton-based Human Motion Prediction](https://mlsanity.com/paper/2311.17408/) (2023)
- [Self-Supervised Human Activity Recognition with Localized Time-Frequency Contrastive Representation Learning](https://mlsanity.com/paper/2209.00990/) (2022)
- [A Novel Skeleton-Based Human Activity Discovery Using Particle Swarm Optimization with Gaussian Mutation](https://mlsanity.com/paper/2201.05314/) (2022)

## Built on by (most cited first, 5 of 5)

- [CHASE: Learning Convex Hull Adaptive Shift for Skeleton-based Multi-Entity Action Recognition](https://mlsanity.com/paper/2410.07153/) (2024)
- [Learning Adaptive Node Selection with External Attention for Human Interaction Recognition](https://mlsanity.com/paper/2507.03936/) (2025)
- [3D Human-Human Interaction Anomaly Detection](https://mlsanity.com/paper/2512.13560/) (2025)
- [STAR: Skeletal Token Alignment and Rearrangement for Interaction Recognition](https://mlsanity.com/paper/2607.17342/) (2026)
- [De-biasing Skeleton-based Action Recognition with Convex Hull Adaptive Shift](https://mlsanity.com/paper/2609.32454/) (2026)

## Abstract (arXiv)

Recognizing interactive actions, including hand-to-hand interaction and human-to-human interaction, has attracted increasing attention for various applications in the field of video analysis and human-robot interaction. Considering the success of graph convolution in modeling topology-aware features from skeleton data, recent methods commonly operate graph convolution on separate entities and use late fusion for interactive action recognition, which can barely model the mutual semantic relationships between pairwise entities. To this end, we propose a mutual excitation graph convolutional network (me-GCN) by stacking mutual excitation graph convolution (me-GC) layers. Specifically, me-GC uses a mutual topology excitation module to firstly extract adjacency matrices from individual entities and then adaptively model the mutual constraints between them. Moreover, me-GC extends the above idea and further uses a mutual feature excitation module to extract and merge deep features from pairwise entities. Compared with graph convolution, our proposed me-GC gradually learns mutual information in each layer and each stage of graph convolution operations. Extensive experiments on a challenging hand-to-hand interaction dataset, i.e., the Assembely101 dataset, and two large-scale human-to-human interaction datasets, i.e., NTU60-Interaction and NTU120-Interaction consistently verify the superiority of our proposed method, which outperforms the state-of-the-art GCN-based and Transformer-based methods.