# Interactive Spatiotemporal Token Attention Network for Skeleton-based General Interactive Action Recognition

arXiv: https://arxiv.org/abs/2307.07469 · HTML: https://mlsanity.com/paper/2307.07469/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 34 references that are in the map.
- First posted on arXiv 14 Jul 2023.

## Builds on (most cited first, 20 of 20)

- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [H2O: Two Hands Manipulating Objects for First Person Interaction Recognition](https://mlsanity.com/paper/2104.11181/) (2021)
- [Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/1904.12659/) (2019)
- [Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition](https://mlsanity.com/paper/1607.07043/) (2016)
- [H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions](https://mlsanity.com/paper/1904.05349/) (2019)
- [View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data](https://mlsanity.com/paper/1703.08274/) (2017)
- [Hierarchically Decomposed Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2208.10741/) (2022)
- [Co-occurrence Feature Learning for Skeleton based Action Recognition using Regularized Deep LSTM Networks](https://mlsanity.com/paper/1603.07772/) (2016)
- [Skeleton-Based Human Action Recognition with Global Context-Aware Attention LSTM Networks](https://mlsanity.com/paper/1707.05740/) (2017)
- [IGFormer: Interaction Graph Transformer for Skeleton-based Human Interaction Recognition](https://mlsanity.com/paper/2207.12100/) (2022)
- [Interaction Relational Network for Mutual Action Recognition](https://mlsanity.com/paper/1910.04963/) (2019)
- [Let's Play for Action: Recognizing Activities of Daily Living by Learning from Life Simulation Video Games](https://mlsanity.com/paper/2107.05617/) (2021)
- [Skeleton-based Action Recognition via Temporal-Channel Aggregation](https://mlsanity.com/paper/2205.15936/) (2022)
- [Robust Event Detection based on Spatio-Temporal Latent Action Unit using Skeletal Information](https://mlsanity.com/paper/2109.02376/) (2021)
- [Non-local Graph Convolutional Network for joint Activity Recognition and Motion Prediction](https://mlsanity.com/paper/2108.01518/) (2021)

## Built on by (most cited first, 23 of 23)

- [An Outlook into the Future of Egocentric Vision](https://mlsanity.com/paper/2308.07123/) (2023)
- [SkateFormer: Skeletal-Temporal Transformer for Human Action Recognition](https://mlsanity.com/paper/2403.09508/) (2024)
- [On the Utility of 3D Hand Poses for Action Recognition](https://mlsanity.com/paper/2403.09805/) (2024)
- [Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognition](https://mlsanity.com/paper/2407.15706/) (2024)
- [Explore Human Parsing Modality for Action Recognition](https://mlsanity.com/paper/2401.02138/) (2024)
- [3D Skeleton-Based Action Recognition: A Review](https://mlsanity.com/paper/2506.00915/) (2025)
- [HDBN: A Novel Hybrid Dual-branch Network for Robust Skeleton-based Action Recognition](https://mlsanity.com/paper/2404.15719/) (2024)
- [Body-Hand Modality Expertized Networks with Cross-attention for Fine-grained Skeleton Action Recognition](https://mlsanity.com/paper/2503.14960/) (2025)
- [A Two-stream Hybrid CNN-Transformer Network for Skeleton-based Human Interaction Recognition](https://mlsanity.com/paper/2401.00409/) (2023)
- [CHASE: Learning Convex Hull Adaptive Shift for Skeleton-based Multi-Entity Action Recognition](https://mlsanity.com/paper/2410.07153/) (2024)
- [Integrating Human Parsing and Pose Network for Human Action Recognition](https://mlsanity.com/paper/2307.07977/) (2023)
- [A Generically Contrastive Spatiotemporal Representation Enhancement for 3D Skeleton Action Recognition](https://mlsanity.com/paper/2312.15144/) (2023)
- [SkelMamba: A State Space Model for Efficient Skeleton Action Recognition of Neurological Disorders](https://mlsanity.com/paper/2411.19544/) (2024)
- [Learning Adaptive Node Selection with External Attention for Human Interaction Recognition](https://mlsanity.com/paper/2507.03936/) (2025)
- [SemiPL: A Semi-supervised Method for Event Sound Source Localization](https://mlsanity.com/paper/2404.19615/) (2024)
- [MaskSem: Semantic-Guided Masking for Learning 3D Hybrid High-Order Motion Representation](https://mlsanity.com/paper/2508.12948/) (2025)
- [TSkel-Mamba: Temporal Dynamic Modeling via State Space Model for Human Skeleton-based Action Recognition](https://mlsanity.com/paper/2512.11503/) (2025)
- [3D Human-Human Interaction Anomaly Detection](https://mlsanity.com/paper/2512.13560/) (2025)
- [BHaRNet: Reliability-Aware Body-Hand Modality Expertized Networks for Fine-grained Skeleton Action Recognition](https://mlsanity.com/paper/2601.00369/) (2026)
- [STAR: Skeletal Token Alignment and Rearrangement for Interaction Recognition](https://mlsanity.com/paper/2607.17342/) (2026)
- [Structure over Depth: A Single-Block Spatio-Temporal Transformer for Multi-Entity Reasoning](https://mlsanity.com/paper/2607.23077/) (2026)
- [One for All: Generalist Foundation Model for Cross-Sensor Skeleton Representation Learning](https://mlsanity.com/paper/2609.07078/) (2026)
- [De-biasing Skeleton-based Action Recognition with Convex Hull Adaptive Shift](https://mlsanity.com/paper/2609.32454/) (2026)

## Abstract (arXiv)

Recognizing interactive action plays an important role in human-robot interaction and collaboration. Previous methods use late fusion and co-attention mechanism to capture interactive relations, which have limited learning capability or inefficiency to adapt to more interacting entities. With assumption that priors of each entity are already known, they also lack evaluations on a more general setting addressing the diversity of subjects. To address these problems, we propose an Interactive Spatiotemporal Token Attention Network (ISTA-Net), which simultaneously model spatial, temporal, and interactive relations. Specifically, our network contains a tokenizer to partition Interactive Spatiotemporal Tokens (ISTs), which is a unified way to represent motions of multiple diverse entities. By extending the entity dimension, ISTs provide better interactive representations. To jointly learn along three dimensions in ISTs, multi-head self-attention blocks integrated with 3D convolutions are designed to capture inter-token correlations. When modeling correlations, a strict entity ordering is usually irrelevant for recognizing interactive actions. To this end, Entity Rearrangement is proposed to eliminate the orderliness in ISTs for interchangeable entities. Extensive experiments on four datasets verify the effectiveness of ISTA-Net by outperforming state-of-the-art methods. Our code is publicly available at https://github.com/Necolizer/ISTA-Net