# BEST: BERT Pre-Training for Sign Language Recognition with Coupling Tokenization

arXiv: https://arxiv.org/abs/2302.05075 · HTML: https://mlsanity.com/paper/2302.05075/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 47 references that are in the map.
- First posted on arXiv 10 Feb 2023.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [ActBERT: Learning Global-Local Video-Text Representations](https://mlsanity.com/paper/2011.07231/) (2020)
- [Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison](https://mlsanity.com/paper/1910.11006/) (2019)
- [MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language](https://mlsanity.com/paper/1812.01053/) (2018)
- [Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation](https://mlsanity.com/paper/1804.06055/) (2018)
- [BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues](https://mlsanity.com/paper/2007.12131/) (2020)
- [AUTSL: A Large Scale Multi-modal Turkish Sign Language Dataset and Baseline Methods](https://mlsanity.com/paper/2008.00932/) (2020)
- [Transferring Cross-domain Knowledge for Video Sign Language Recognition](https://mlsanity.com/paper/2003.03703/) (2020)
- [SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognition](https://mlsanity.com/paper/2110.05382/) (2021)
- [Global-local Enhancement Network for NMFs-aware Sign Language Recognition](https://mlsanity.com/paper/2008.10428/) (2020)
- [Pose-based Sign Language Recognition using GCN and BERT](https://mlsanity.com/paper/2012.00781/) (2020)
- [Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamics](https://mlsanity.com/paper/2007.12287/) (2020)
- [OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languages](https://mlsanity.com/paper/2110.05877/) (2021)
- [Skeletor: Skeletal Transformers for Robust Body-Pose Estimation](https://mlsanity.com/paper/2104.11712/) (2021)

## Built on by (most cited first, 30 of 35)

- [Uni-Sign: Toward Unified Sign Language Understanding at Scale](https://mlsanity.com/paper/2501.15187/) (2025)
- [Natural Language-Assisted Sign Language Recognition](https://mlsanity.com/paper/2303.12080/) (2023)
- [Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions](https://mlsanity.com/paper/2501.04437/) (2025)
- [Scaling up Multimodal Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2408.08544/) (2024)
- [Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator](https://mlsanity.com/paper/2411.17799/) (2024)
- [A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars](https://mlsanity.com/paper/2401.04730/) (2024)
- [Discrete to Continuous: Generating Smooth Transition Poses from Sign Language Observation](https://mlsanity.com/paper/2411.16810/) (2024)
- [SignRep: Enhancing Self-Supervised Sign Representations](https://mlsanity.com/paper/2503.08529/) (2025)
- [MASA: Motion-aware Masked Autoencoder with Semantic Alignment for Sign Language Recognition](https://mlsanity.com/paper/2405.20666/) (2024)
- [Geo-Sign: Hyperbolic Contrastive Regularisation for Geometrically Aware Sign Language Translation](https://mlsanity.com/paper/2506.00129/) (2025)
- [Self-Supervised Representation Learning with Spatial-Temporal Consistency for Sign Language Recognition](https://mlsanity.com/paper/2406.10501/) (2024)
- [Learnt Contrastive Concept Embeddings for Sign Recognition](https://mlsanity.com/paper/2308.09515/) (2023)
- [Dynamic Spatial-Temporal Aggregation for Skeleton-Aware Sign Language Recognition](https://mlsanity.com/paper/2403.12519/) (2024)
- [SignX: Continuous Sign Recognition in Compact Pose-Rich Latent Space](https://mlsanity.com/paper/2504.16315/) (2025)
- [MaDiS: Taming Masked Diffusion Language Models for Sign Language Generation](https://mlsanity.com/paper/2601.19577/) (2026)
- [Logos as a Well-Tempered Pre-train for Sign Language Recognition](https://mlsanity.com/paper/2505.10481/) (2025)
- [M3T: Discrete Multi-Modal Motion Tokens for Sign Language Production](https://mlsanity.com/paper/2603.23617/) (2026)
- [Exploiting Spatial-Temporal Context for Interacting Hand Reconstruction on Monocular RGB Video](https://mlsanity.com/paper/2308.04074/) (2023)
- [Cross-Modal Consistency Learning for Sign Language Recognition](https://mlsanity.com/paper/2503.12485/) (2025)
- [Fine-Tuning Video Transformers for Word-Level Bangla Sign Language: A Comparative Analysis for Classification Tasks](https://mlsanity.com/paper/2506.04367/) (2025)
- [SignBart -- New approach with the skeleton sequence for Isolated Sign language Recognition](https://mlsanity.com/paper/2506.21592/) (2025)
- [Motion is the Choreographer: Learning Latent Pose Dynamics for Seamless Sign Language Generation](https://mlsanity.com/paper/2508.04049/) (2025)
- [FR-GESTURE: An RGBD Dataset For Gesture-based Human-Robot Interaction In First Responder Operations](https://mlsanity.com/paper/2602.17573/) (2026)
- [SignAgent: Agentic LLMs for Linguistically-Grounded Sign Language Annotation and Dataset Curation](https://mlsanity.com/paper/2603.19059/) (2026)
- [Stack Transformer Based Spatial-Temporal Attention Model for Dynamic Sign Language and Fingerspelling Recognition](https://mlsanity.com/paper/2503.16855/) (2025)
- [Generative Sign-description Prompts with Multi-positive Contrastive Learning for Sign Language Recognition](https://mlsanity.com/paper/2505.02304/) (2025)
- [Hybrid Autoregressive-Diffusion Model for Real-Time Sign Language Production](https://mlsanity.com/paper/2507.09105/) (2025)
- [SSL-SLR: Self-Supervised Representation Learning for Sign Language Recognition](https://mlsanity.com/paper/2509.05188/) (2025)
- [SegSLR: Promptable Video Segmentation for Isolated Sign Language Recognition](https://mlsanity.com/paper/2509.10710/) (2025)
- [LA-Sign: Looped Transformers with Geometry-aware Alignment for Skeleton-based Sign Language Recognition](https://mlsanity.com/paper/2603.29057/) (2026)

## Abstract (arXiv)

In this work, we are dedicated to leveraging the BERT pre-training success and modeling the domain-specific statistics to fertilize the sign language recognition~(SLR) model. Considering the dominance of hand and body in sign language expression, we organize them as pose triplet units and feed them into the Transformer backbone in a frame-wise manner. Pre-training is performed via reconstructing the masked triplet unit from the corrupted input sequence, which learns the hierarchical correlation context cues among internal and external triplet units. Notably, different from the highly semantic word token in BERT, the pose unit is a low-level signal originally located in continuous space, which prevents the direct adoption of the BERT cross-entropy objective. To this end, we bridge this semantic gap via coupling tokenization of the triplet unit. It adaptively extracts the discrete pseudo label from the pose triplet unit, which represents the semantic gesture/body state. After pre-training, we fine-tune the pre-trained encoder on the downstream SLR task, jointly with the newly added task-specific layer. Extensive experiments are conducted to validate the effectiveness of our proposed method, achieving new state-of-the-art performance on all four benchmarks with a notable gain.