# Ske2Grid: Skeleton-to-Grid Representation Learning for Action Recognition

arXiv: https://arxiv.org/abs/2308.07571 · HTML: https://mlsanity.com/paper/2308.07571/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 38 references that are in the map.
- First posted on arXiv 15 Aug 2023.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [Revisiting Skeleton-based Action Recognition](https://mlsanity.com/paper/2104.13586/) (2021)
- [FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding](https://mlsanity.com/paper/2004.06704/) (2020)
- [Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/1904.12659/) (2019)
- [Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN](https://mlsanity.com/paper/1803.04831/) (2018)
- [An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data](https://mlsanity.com/paper/1611.06067/) (2016)
- [A New Representation of Skeleton Sequences for 3D Action Recognition](https://mlsanity.com/paper/1703.03492/) (2017)
- [2D/3D Pose Estimation and Action Recognition using Multitask Deep Learning](https://mlsanity.com/paper/1802.09232/) (2018)
- [Skeleton-Based Action Recognition with Multi-Stream Adaptive Graph Convolutional Networks](https://mlsanity.com/paper/1912.06971/) (2019)
- [Skeleton-based Action Recognition via Spatial and Temporal Transformer Networks](https://mlsanity.com/paper/2008.07404/) (2020)
- [PYSKL: Towards Good Practices for Skeleton Action Recognition](https://mlsanity.com/paper/2205.09443/) (2022)
- [Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching](https://mlsanity.com/paper/1911.04131/) (2019)
- [Symbiotic Graph Neural Networks for 3D Skeleton-based Human Action Recognition and Motion Prediction](https://mlsanity.com/paper/1910.02212/) (2019)
- [Hypergraph Transformer for Skeleton-based Action Recognition](https://mlsanity.com/paper/2211.09590/) (2022)
- [DG-STGCN: Dynamic Spatial-Temporal Modeling for Skeleton-based Action Recognition](https://mlsanity.com/paper/2210.05895/) (2022)
- [Fusing Higher-order Features in Graph Neural Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/2105.01563/) (2021)
- [Skeleton-Based Action Recognition with Synchronous Local and Non-local Spatio-temporal Learning and Frequency Attention](https://mlsanity.com/paper/1811.04237/) (2018)
- [3D Human Pose Lifting with Grid Convolution](https://mlsanity.com/paper/2302.08760/) (2023)

## Built on by (most cited first, 7 of 7)

- [SkateFormer: Skeletal-Temporal Transformer for Human Action Recognition](https://mlsanity.com/paper/2403.09508/) (2024)
- [LLMs are Good Action Recognizers](https://mlsanity.com/paper/2404.00532/) (2024)
- [Bridging the Skeleton-Text Modality Gap: Diffusion-Powered Modality Alignment for Zero-shot Skeleton-based Action Recognition](https://mlsanity.com/paper/2411.10745/) (2024)
- [Frequency-Enhanced Diffusion Models: Curriculum-Guided Semantic Alignment for Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2604.09063/) (2026)
- [SkelMamba: A State Space Model for Efficient Skeleton Action Recognition of Neurological Disorders](https://mlsanity.com/paper/2411.19544/) (2024)
- [CascadeFormer: A Family of Two-stage Cascading Transformers for Skeleton-based Human Action Recognition](https://mlsanity.com/paper/2509.00692/) (2025)
- [Affinity Contrastive Learning for Skeleton-based Human Activity Understanding](https://mlsanity.com/paper/2601.16694/) (2026)

## Abstract (arXiv)

This paper presents Ske2Grid, a new representation learning framework for improved skeleton-based action recognition. In Ske2Grid, we define a regular convolution operation upon a novel grid representation of human skeleton, which is a compact image-like grid patch constructed and learned through three novel designs. Specifically, we propose a graph-node index transform (GIT) to construct a regular grid patch through assigning the nodes in the skeleton graph one by one to the desired grid cells. To ensure that GIT is a bijection and enrich the expressiveness of the grid representation, an up-sampling transform (UPT) is learned to interpolate the skeleton graph nodes for filling the grid patch to the full. To resolve the problem when the one-step UPT is aggressive and further exploit the representation capability of the grid patch with increasing spatial size, a progressive learning strategy (PLS) is proposed which decouples the UPT into multiple steps and aligns them to multiple paired GITs through a compact cascaded design learned progressively. We construct networks upon prevailing graph convolution networks and conduct experiments on six mainstream skeleton-based action recognition datasets. Experiments show that our Ske2Grid significantly outperforms existing GCN-based solutions under different benchmark settings, without bells and whistles. Code and models are available at https://github.com/OSVAI/Ske2Grid