# How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language

arXiv: https://arxiv.org/abs/2008.08143 · HTML: https://mlsanity.com/paper/2008.08143/

## Key facts

- Cited by 173 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Aug 2020.

## Built on by (most cited first, 30 of 173)

- [One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer](https://mlsanity.com/paper/2303.16160/) (2023)
- [Retrieving Multimodal Information for Augmented Generation: A Survey](https://mlsanity.com/paper/2303.10868/) (2023)
- [A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation](https://mlsanity.com/paper/2203.04287/) (2022)
- [LLMs are Good Sign Language Translators](https://mlsanity.com/paper/2404.00925/) (2024)
- [YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpus](https://mlsanity.com/paper/2306.15162/) (2023)
- [Open-Domain Sign Language Translation Learned from Online Video](https://mlsanity.com/paper/2205.12870/) (2022)
- [SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2305.04868/) (2023)
- [Uni-Sign: Toward Unified Sign Language Understanding at Scale](https://mlsanity.com/paper/2501.15187/) (2025)
- [Neural Sign Actors: A diffusion model for 3D sign language production from text](https://mlsanity.com/paper/2312.02702/) (2023)
- [Towards Privacy-Aware Sign Language Translation at Scale](https://mlsanity.com/paper/2402.09611/) (2024)
- [Sign Language Translation from Instructional Videos](https://mlsanity.com/paper/2304.06371/) (2023)
- [YouTube-SL-25: A Large-Scale, Open-Domain Multilingual Sign Language Parallel Corpus](https://mlsanity.com/paper/2407.11144/) (2024)
- [CiCo: Domain-Aware Sign Language Retrieval via Cross-Lingual Contrastive Learning](https://mlsanity.com/paper/2303.12793/) (2023)
- [Gloss-Free End-to-End Sign Language Translation](https://mlsanity.com/paper/2305.12876/) (2023)
- [DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teachers](https://mlsanity.com/paper/2503.14405/) (2025)
- [T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Text](https://mlsanity.com/paper/2406.07119/) (2024)
- [SignAvatars: A Large-scale 3D Sign Language Holistic Motion Dataset and Benchmark](https://mlsanity.com/paper/2310.20436/) (2023)
- [BBC-Oxford British Sign Language Dataset](https://mlsanity.com/paper/2111.03635/) (2021)
- [Sign Language Production: A Review](https://mlsanity.com/paper/2103.15910/) (2021)
- [Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues](https://mlsanity.com/paper/2501.09754/) (2025)
- [SignDiff: Diffusion Model for American Sign Language Production](https://mlsanity.com/paper/2308.16082/) (2023)
- [Factorized Learning Assisted with Large Language Model for Gloss-free Sign Language Translation](https://mlsanity.com/paper/2403.12556/) (2024)
- [C2RL: Content and Context Representation Learning for Gloss-free Sign Language Translation and Retrieval](https://mlsanity.com/paper/2408.09949/) (2024)
- [Sign Language Video Retrieval with Free-Form Textual Queries](https://mlsanity.com/paper/2201.02495/) (2022)
- [Vision+X: A Survey on Multimodal Learning in the Light of Data](https://mlsanity.com/paper/2210.02884/) (2022)
- [Beyond FVD: Enhanced Evaluation Metrics for Video Generation Quality](https://mlsanity.com/paper/2410.05203/) (2024)
- [SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Prediction](https://mlsanity.com/paper/2411.16765/) (2024)
- [GUAVA: Generalizable Upper Body 3D Gaussian Avatar](https://mlsanity.com/paper/2505.03351/) (2025)
- [Ham2Pose: Animating Sign Language Notation into Pose Sequences](https://mlsanity.com/paper/2211.13613/) (2022)
- [Scaling up Multimodal Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2408.08544/) (2024)

## Abstract (arXiv)

One of the factors that have hindered progress in the areas of sign language recognition, translation, and production is the absence of large annotated datasets. Towards this end, we introduce How2Sign, a multimodal and multiview continuous American Sign Language (ASL) dataset, consisting of a parallel corpus of more than 80 hours of sign language videos and a set of corresponding modalities including speech, English transcripts, and depth. A three-hour subset was further recorded in the Panoptic studio enabling detailed 3D pose estimation. To evaluate the potential of How2Sign for real-world impact, we conduct a study with ASL signers and show that synthesized videos using our dataset can indeed be understood. The study further gives insights on challenges that computer vision should address in order to make progress in this field. Dataset website: http://how2sign.github.io/