# Uni-Sign: Toward Unified Sign Language Understanding at Scale

arXiv: https://arxiv.org/abs/2501.15187 · HTML: https://mlsanity.com/paper/2501.15187/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 72 references that are in the map.
- First posted on arXiv 25 Jan 2025.

## Builds on (most cited first, 30 of 56)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation](https://mlsanity.com/paper/2003.13830/) (2020)
- [Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison](https://mlsanity.com/paper/1910.11006/) (2019)
- [Improving Sign Language Translation with Monolingual Data by Sign Back-Translation](https://mlsanity.com/paper/2105.12397/) (2021)
- [Whole-Body Human Pose Estimation in the Wild](https://mlsanity.com/paper/2007.11858/) (2020)
- [RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose](https://mlsanity.com/paper/2303.07399/) (2023)
- [How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language](https://mlsanity.com/paper/2008.08143/) (2020)
- [A Unified Sequence Interface for Vision Tasks](https://mlsanity.com/paper/2206.07669/) (2022)
- [VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks](https://mlsanity.com/paper/2406.08394/) (2024)
- [Two-Stream Network for Sign Language Recognition and Translation](https://mlsanity.com/paper/2211.01367/) (2022)
- [MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language](https://mlsanity.com/paper/1812.01053/) (2018)
- [Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining](https://mlsanity.com/paper/2307.14768/) (2023)
- [Skeleton Aware Multi-modal Sign Language Recognition](https://mlsanity.com/paper/2103.08833/) (2021)
- [FunASR: A Fundamental End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2305.11013/) (2023)
- [Visual Alignment Constraint for Continuous Sign Language Recognition](https://mlsanity.com/paper/2104.02330/) (2021)
- [A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation](https://mlsanity.com/paper/2203.04287/) (2022)
- [LLMs are Good Sign Language Translators](https://mlsanity.com/paper/2404.00925/) (2024)
- [YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpus](https://mlsanity.com/paper/2306.15162/) (2023)
- [Open-Domain Sign Language Translation Learned from Online Video](https://mlsanity.com/paper/2205.12870/) (2022)
- [Including Signed Languages in Natural Language Processing](https://mlsanity.com/paper/2105.05222/) (2021)
- [Neural Sign Language Translation based on Human Keypoint Estimation](https://mlsanity.com/paper/1811.11436/) (2018)

## Built on by (most cited first, 30 of 56)

- [Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation](https://mlsanity.com/paper/2505.15438/) (2025)
- [Geo-Sign: Hyperbolic Contrastive Regularisation for Geometrically Aware Sign Language Translation](https://mlsanity.com/paper/2506.00129/) (2025)
- [Gloss-Free Sign Language Translation: An Unbiased Evaluation of Progress in the Field](https://mlsanity.com/paper/2603.13240/) (2026)
- [Isharah: A Large-Scale Multi-Scene Dataset for Continuous Sign Language Recognition](https://mlsanity.com/paper/2506.03615/) (2025)
- [Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment](https://mlsanity.com/paper/2512.08040/) (2025)
- [Body-Hand Modality Expertized Networks with Cross-attention for Fine-grained Skeleton Action Recognition](https://mlsanity.com/paper/2503.14960/) (2025)
- [SignX: Continuous Sign Recognition in Compact Pose-Rich Latent Space](https://mlsanity.com/paper/2504.16315/) (2025)
- [MaDiS: Taming Masked Diffusion Language Models for Sign Language Generation](https://mlsanity.com/paper/2601.19577/) (2026)
- [Using Sign Language Production as Data Augmentation to enhance Sign Language Translation](https://mlsanity.com/paper/2506.09643/) (2025)
- [SONAR-SLT: Multilingual Sign Language Translation via Language-Agnostic Sentence Embedding Supervision](https://mlsanity.com/paper/2510.19398/) (2025)
- [Logos as a Well-Tempered Pre-train for Sign Language Recognition](https://mlsanity.com/paper/2505.10481/) (2025)
- [RVLF: A Reinforcing Vision-Language Framework for Gloss-Free Sign Language Translation](https://mlsanity.com/paper/2512.07273/) (2025)
- [SignSparK: Efficient Multilingual Sign Language Production via Sparse Keyframe Learning](https://mlsanity.com/paper/2603.10446/) (2026)
- [M3T: Discrete Multi-Modal Motion Tokens for Sign Language Production](https://mlsanity.com/paper/2603.23617/) (2026)
- [Sign Language Question Answering: A New Task, Benchmark, and Baseline for Sign Language Understanding](https://mlsanity.com/paper/2607.27826/) (2026)
- [Sign Spotting Disambiguation using Large Language Models](https://mlsanity.com/paper/2507.03703/) (2025)
- [SAGE: Segment-Aware Gloss-Free Encoding for Token-Efficient Sign Language Translation](https://mlsanity.com/paper/2507.09266/) (2025)
- [Towards Inclusive Communication: A Unified Framework for Generating Spoken Language from Sign, Lip, and Audio](https://mlsanity.com/paper/2508.20476/) (2025)
- [SignBind-LLM: Multi-Stage Modality Fusion for Sign Language Translation](https://mlsanity.com/paper/2509.00030/) (2025)
- [SignAgent: Agentic LLMs for Linguistically-Grounded Sign Language Annotation and Dataset Curation](https://mlsanity.com/paper/2603.19059/) (2026)
- [SIGNET: Motion-Level Knowledge Transfer for Cross-Language Sign Language Translation](https://mlsanity.com/paper/2606.28626/) (2026)
- [BackTranslation2.0 -- A Linguistically Motivated Metric to Assess Sign Language Production](https://mlsanity.com/paper/2606.28673/) (2026)
- [Investigating Multimodal Informativity under Different Partner Visibility Conditions in Video-Mediated Dialogue](https://mlsanity.com/paper/2608.08915/) (2026)
- [Stack Transformer Based Spatial-Temporal Attention Model for Dynamic Sign Language and Fingerspelling Recognition](https://mlsanity.com/paper/2503.16855/) (2025)
- [DESign: Dynamic Context-Aware Convolution and Efficient Subnet Regularization for Continuous Sign Language Recognition](https://mlsanity.com/paper/2507.03339/) (2025)
- [Skeleton-based sign language recognition using a dual-stream spatio-temporal dynamic graph convolutional network](https://mlsanity.com/paper/2509.08661/) (2025)
- [Seeing, Signing, and Saying: A Vision-Language Model-Assisted Pipeline for Sign Language Data Acquisition and Curation from Social Media](https://mlsanity.com/paper/2510.25413/) (2025)
- [Data-Efficient American Sign Language Recognition via Few-Shot Prototypical Networks](https://mlsanity.com/paper/2512.10562/) (2025)
- [BHaRNet: Reliability-Aware Body-Hand Modality Expertized Networks for Fine-grained Skeleton Action Recognition](https://mlsanity.com/paper/2601.00369/) (2026)
- [FEA-SLT: A Gloss-Free End-to-End Framework for Facial-Expression-Aware Sign Language Translation](https://mlsanity.com/paper/2601.03549/) (2026)

## Abstract (arXiv)

Sign language pre-training has gained increasing attention for its ability to enhance performance across various sign language understanding (SLU) tasks. However, existing methods often suffer from a gap between pre-training and fine-tuning, leading to suboptimal results. To address this, we propose Uni-Sign, a unified pre-training framework that eliminates the gap between pre-training and downstream SLU tasks through a large-scale generative pre-training strategy and a novel fine-tuning paradigm. First, we introduce CSL-News, a large-scale Chinese Sign Language (CSL) dataset containing 1,985 hours of video paired with textual annotations, which enables effective large-scale pre-training. Second, Uni-Sign unifies SLU tasks by treating downstream tasks as a single sign language translation (SLT) task during fine-tuning, ensuring seamless knowledge transfer between pre-training and fine-tuning. Furthermore, we incorporate a prior-guided fusion (PGF) module and a score-aware sampling strategy to efficiently fuse pose and RGB information, addressing keypoint inaccuracies and improving computational efficiency. Extensive experiments across multiple SLU benchmarks demonstrate that Uni-Sign achieves state-of-the-art performance across multiple downstream SLU tasks. Dataset and code are available at github.com/ZechengLi19/Uni-Sign.