# Memory Attention Networks for Skeleton-based Action Recognition

arXiv: https://arxiv.org/abs/1804.08254 · HTML: https://mlsanity.com/paper/1804.08254/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Apr 2018.

## Built on by (most cited first, 30 of 36)

- [Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimation](https://mlsanity.com/paper/2103.14304/) (2021)
- [FG-CLIP: Fine-Grained Visual and Textual Alignment](https://mlsanity.com/paper/2505.05071/) (2025)
- [UNIK: A Unified Framework for Real-world Skeleton-based Action Recognition](https://mlsanity.com/paper/2107.08580/) (2021)
- [Uplift and Upsample: Efficient 3D Human Pose Estimation with Uplifting Transformers](https://mlsanity.com/paper/2210.06110/) (2022)
- [DirecFormer: A Directed Attention in Transformer Approach to Robust Action Recognition](https://mlsanity.com/paper/2203.10233/) (2022)
- [Expansion-Squeeze-Excitation Fusion Network for Elderly Activity Recognition](https://mlsanity.com/paper/2112.10992/) (2021)
- [Fusing Higher-order Features in Graph Neural Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/2105.01563/) (2021)
- [Focal and Global Spatial-Temporal Transformer for Skeleton-based Action Recognition](https://mlsanity.com/paper/2210.02693/) (2022)
- [Spatio-Temporal Tuples Transformer for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2201.02849/) (2022)
- [Spatiotemporal Decouple-and-Squeeze Contrastive Learning for Semi-Supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2302.02316/) (2023)
- [Skeleton-Based Mutually Assisted Interacted Object Localization and Human Action Recognition](https://mlsanity.com/paper/2110.14994/) (2021)
- [Multimodal Transformer for Nursing Activity Recognition](https://mlsanity.com/paper/2204.04564/) (2022)
- [Dynamic Hypergraph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2112.10570/) (2021)
- [STAR: Sparse Transformer-based Action Recognition](https://mlsanity.com/paper/2107.07089/) (2021)
- [A Methodological and Structural Review of Hand Gesture Recognition Across Diverse Data Modalities](https://mlsanity.com/paper/2408.05436/) (2024)
- [Skeleton Sequence and RGB Frame Based Multi-Modality Feature Fusion Network for Action Recognition](https://mlsanity.com/paper/2202.11374/) (2022)
- [3D Skeleton-Based Action Recognition: A Review](https://mlsanity.com/paper/2506.00915/) (2025)
- [Invertible Attention](https://mlsanity.com/paper/2106.09003/) (2021)
- [Towards To-a-T Spatio-Temporal Focus for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2202.02314/) (2022)
- [Spatial-temporal Transformer-guided Diffusion based Data Augmentation for Efficient Skeleton-based Action Recognition](https://mlsanity.com/paper/2302.13434/) (2023)
- [Pyramid Pixel Context Adaption Network for Medical Image Classification with Supervised Contrastive Learning](https://mlsanity.com/paper/2303.01917/) (2023)
- [A Novel Skeleton-Based Human Activity Discovery Using Particle Swarm Optimization with Gaussian Mutation](https://mlsanity.com/paper/2201.05314/) (2022)
- [Strengthening Skeletal Action Recognizers via Leveraging Temporal Patterns](https://mlsanity.com/paper/2205.14405/) (2022)
- [Weakly-Supervised Temporal Action Localization with Bidirectional Semantic Consistency Constraint](https://mlsanity.com/paper/2304.12616/) (2023)
- [Enhancing Action Recognition from Low-Quality Skeleton Data via Part-Level Knowledge Distillation](https://mlsanity.com/paper/2404.18206/) (2024)
- [Signal-SGN: A Spiking Graph Convolutional Network for Skeletal Action Recognition via Learning Temporal-Frequency Dynamics](https://mlsanity.com/paper/2408.01701/) (2024)
- [Border-SegGCN: Improving Semantic Segmentation by Refining the Border Outline using Graph Convolutional Network](https://mlsanity.com/paper/2109.05353/) (2021)
- [Skeletal Human Action Recognition using Hybrid Attention based Graph Convolutional Network](https://mlsanity.com/paper/2207.05493/) (2022)
- [Evolving Skeletons: Motion Dynamics in Action Recognition](https://mlsanity.com/paper/2501.02593/) (2025)
- [Development of Human Motion Prediction Strategy using Inception Residual Block](https://mlsanity.com/paper/2108.04001/) (2021)

## Abstract (arXiv)

Skeleton-based action recognition task is entangled with complex spatio-temporal variations of skeleton joints, and remains challenging for Recurrent Neural Networks (RNNs). In this work, we propose a temporal-then-spatial recalibration scheme to alleviate such complex variations, resulting in an end-to-end Memory Attention Networks (MANs) which consist of a Temporal Attention Recalibration Module (TARM) and a Spatio-Temporal Convolution Module (STCM). Specifically, the TARM is deployed in a residual learning module that employs a novel attention learning network to recalibrate the temporal attention of frames in a skeleton sequence. The STCM treats the attention calibrated skeleton joint sequences as images and leverages the Convolution Neural Networks (CNNs) to further model the spatial and temporal information of skeleton data. These two modules (TARM and STCM) seamlessly form a single network architecture that can be trained in an end-to-end fashion. MANs significantly boost the performance of skeleton-based action recognition and achieve the best results on four challenging benchmark datasets: NTU RGB+D, HDM05, SYSU-3D and UT-Kinect.