# MASA: Motion-aware Masked Autoencoder with Semantic Alignment for Sign Language Recognition

arXiv: https://arxiv.org/abs/2405.20666 · HTML: https://mlsanity.com/paper/2405.20666/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 73 references that are in the map.
- First posted on arXiv 31 May 2024.

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison](https://mlsanity.com/paper/1910.11006/) (2019)
- [VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examples](https://mlsanity.com/paper/2103.05905/) (2021)
- [Contrastive Masked Autoencoders are Stronger Vision Learners](https://mlsanity.com/paper/2207.13532/) (2022)
- [MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language](https://mlsanity.com/paper/1812.01053/) (2018)
- [Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation](https://mlsanity.com/paper/1804.06055/) (2018)
- [BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues](https://mlsanity.com/paper/2007.12131/) (2020)
- [Boosting Self-Supervised Learning via Knowledge Transfer](https://mlsanity.com/paper/1805.00385/) (2018)
- [Siamese Image Modeling for Self-Supervised Vision Representation Learning](https://mlsanity.com/paper/2206.01204/) (2022)
- [AUTSL: A Large Scale Multi-modal Turkish Sign Language Dataset and Baseline Methods](https://mlsanity.com/paper/2008.00932/) (2020)
- [Transferring Cross-domain Knowledge for Video Sign Language Recognition](https://mlsanity.com/paper/2003.03703/) (2020)
- [SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognition](https://mlsanity.com/paper/2110.05382/) (2021)
- [SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2305.04868/) (2023)
- [Continuous Sign Language Recognition with Correlation Network](https://mlsanity.com/paper/2303.03202/) (2023)

## Built on by (most cited first, 13 of 13)

- [Uni-Sign: Toward Unified Sign Language Understanding at Scale](https://mlsanity.com/paper/2501.15187/) (2025)
- [C2RL: Content and Context Representation Learning for Gloss-free Sign Language Translation and Retrieval](https://mlsanity.com/paper/2408.09949/) (2024)
- [Scaling up Multimodal Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2408.08544/) (2024)
- [Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator](https://mlsanity.com/paper/2411.17799/) (2024)
- [SMC++: Masked Learning of Unsupervised Video Semantic Compression](https://mlsanity.com/paper/2406.04765/) (2024)
- [Self-Supervised Representation Learning with Spatial-Temporal Consistency for Sign Language Recognition](https://mlsanity.com/paper/2406.10501/) (2024)
- [MaDiS: Taming Masked Diffusion Language Models for Sign Language Generation](https://mlsanity.com/paper/2601.19577/) (2026)
- [Cross-Modal Consistency Learning for Sign Language Recognition](https://mlsanity.com/paper/2503.12485/) (2025)
- [Generative Sign-description Prompts with Multi-positive Contrastive Learning for Sign Language Recognition](https://mlsanity.com/paper/2505.02304/) (2025)
- [SignMAE: Segmentation-Driven Self-Supervised Learning for Sign Language Recognition](https://mlsanity.com/paper/2605.02094/) (2026)
- [JSL-DC: A Word-Level Japanese Sign Language Dataset with Linguist-Derived Descriptions for Distinguishing Confusable Signs](https://mlsanity.com/paper/2608.18412/) (2026)
- [SignSeek: Learning Transferable Representations for Sign Dictionary Retrieval](https://mlsanity.com/paper/2609.03695/) (2026)
- [SignDino: Self-Supervised Sign Language Representation Learning via Temporal-Axis Self-Distillation](https://mlsanity.com/paper/2609.06296/) (2026)

## Abstract (arXiv)

Sign language recognition (SLR) has long been plagued by insufficient model representation capabilities. Although current pre-training approaches have alleviated this dilemma to some extent and yielded promising performance by employing various pretext tasks on sign pose data, these methods still suffer from two primary limitations: 1) Explicit motion information is usually disregarded in previous pretext tasks, leading to partial information loss and limited representation capability. 2) Previous methods focus on the local context of a sign pose sequence, without incorporating the guidance of the global meaning of lexical signs. To this end, we propose a Motion-Aware masked autoencoder with Semantic Alignment (MASA) that integrates rich motion cues and global semantic information in a self-supervised learning paradigm for SLR. Our framework contains two crucial components, i.e., a motion-aware masked autoencoder (MA) and a momentum semantic alignment module (SA). Specifically, in MA, we introduce an autoencoder architecture with a motion-aware masked strategy to reconstruct motion residuals of masked frames, thereby explicitly exploring dynamic motion cues among sign pose sequences. Moreover, in SA, we embed our framework with global semantic awareness by aligning the embeddings of different augmented samples from the input sequence in the shared latent space. In this way, our framework can simultaneously learn local motion cues and global semantic features for comprehensive sign language representation. Furthermore, we conduct extensive experiments to validate the effectiveness of our method, achieving new state-of-the-art performance on four public benchmarks.