# FLEURS-ASL: Including American Sign Language in Massively Multilingual Multitask Evaluation

arXiv: https://arxiv.org/abs/2408.13585 · HTML: https://mlsanity.com/paper/2408.13585/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 49 references that are in the map.
- First posted on arXiv 24 Aug 2024.

## Builds on (most cited first, 30 of 30)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [Scaling Up Models and Data with textttt5x and textttseqio](https://mlsanity.com/paper/2203.17189/) (2022)
- [How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language](https://mlsanity.com/paper/2008.08143/) (2020)
- [Learning Compact Metrics for MT](https://mlsanity.com/paper/2110.06341/) (2021)
- [Evaluating Discourse Phenomena in Neural Machine Translation](https://mlsanity.com/paper/1711.00513/) (2017)
- [YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpus](https://mlsanity.com/paper/2306.15162/) (2023)
- [A Benchmark for Learning to Translate a New Language from One Grammar Book](https://mlsanity.com/paper/2309.16575/) (2023)
- [Open-Domain Sign Language Translation Learned from Online Video](https://mlsanity.com/paper/2205.12870/) (2022)
- [Building Machine Translation Systems for the Next Thousand Languages](https://mlsanity.com/paper/2205.03983/) (2022)
- [Including Signed Languages in Natural Language Processing](https://mlsanity.com/paper/2105.05222/) (2021)
- [The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English](https://mlsanity.com/paper/1902.01382/) (2019)
- [SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognition](https://mlsanity.com/paper/2110.05382/) (2021)
- [Towards Privacy-Aware Sign Language Translation at Scale](https://mlsanity.com/paper/2402.09611/) (2024)
- [Aligning Subtitles in Sign Language Videos](https://mlsanity.com/paper/2105.02877/) (2021)
- [Evaluating the Immediate Applicability of Pose Estimation for Sign Language Recognition](https://mlsanity.com/paper/2104.10166/) (2021)
- [Evaluating Gender Bias in Speech Translation](https://mlsanity.com/paper/2010.14465/) (2020)
- [SignMusketeers: An Efficient Multi-Stream Approach for Sign Language Translation at Scale](https://mlsanity.com/paper/2406.06907/) (2024)
- [JWSign: A Highly Multilingual Corpus of Bible Translations for more Diversity in Sign Language Processing](https://mlsanity.com/paper/2311.10174/) (2023)
- [Is context all you need? Scaling Neural Sign Language Translation to Large Domains of Discourse](https://mlsanity.com/paper/2308.09622/) (2023)

## Built on by (most cited first, 10 of 10)

- [Multilingual Gloss-free Sign Language Translation: Towards Building a Sign Language Foundation Model](https://mlsanity.com/paper/2505.24355/) (2025)
- [Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment](https://mlsanity.com/paper/2512.08040/) (2025)
- [2M-BELEBELE: Highly Multilingual Speech and American Sign Language Comprehension Dataset](https://mlsanity.com/paper/2412.08274/) (2024)
- [Bootstrapping Sign Language Annotations with Sign Language Models](https://mlsanity.com/paper/2604.07606/) (2026)
- [The TUB Sign Language Corpus Collection](https://mlsanity.com/paper/2508.05374/) (2025)
- [Beyond a Single Reference: Training and Evaluation with Paraphrases in Sign Language Translation](https://mlsanity.com/paper/2601.21128/) (2026)
- [Targeted Linguistic Analysis of Sign Language Models with Minimal Translation Pairs](https://mlsanity.com/paper/2604.27232/) (2026)
- [SignMatch: Matching Dictionary Signs to Continuous Sign Language Video](https://mlsanity.com/paper/2609.01886/) (2026)
- [SignDino: Self-Supervised Sign Language Representation Learning via Temporal-Axis Self-Distillation](https://mlsanity.com/paper/2609.06296/) (2026)
- [SignGPT: Toward LLM-Mediated Sign Language Interaction through Gloss-Free Translation and Generation](https://mlsanity.com/paper/2609.21709/) (2026)

## Abstract (arXiv)

Sign language translation has historically been peripheral to mainstream machine translation research. In order to help converge the fields, we introduce FLEURS-ASL, an extension of the multiway parallel benchmarks FLORES (for text) and FLEURS (for speech) to support their first sign language (as video), American Sign Language, translated by 5 Certified Deaf Interpreters. FLEURS-ASL can be used to evaluate a variety of tasks -- primarily sentence- and discourse-level translation -- between ASL and 200 other languages as text, or 102 languages as speech. We provide baselines for tasks from ASL to English text using a unified modeling approach that incorporates timestamp tokens and previous text tokens in a 34-second context window, trained on random video clips from YouTube-ASL. This model meets or exceeds the performance of phrase-level baselines while supporting a multitude of new tasks. We also use FLEURS-ASL to show that multimodal frontier models have virtually no understanding of ASL, underscoring the importance of including sign languages in standard evaluation suites.