mlsanityData to 14 Oct 2026

FLEURS-ASL: Including American Sign Language in Massively Multilingual Multitask Evaluation

arXiv 2408.13585 ↗First posted 24 Aug 2024cs.CL cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Sign language translation has historically been peripheral to mainstream machine translation research. In order to help converge the fields, we introduce FLEURS-ASL, an extension of the multiway parallel benchmarks FLORES (for text) and FLEURS (for speech) to support their first sign language (as video), American Sign Language, translated by 5 Certified Deaf Interpreters. FLEURS-ASL can be used to evaluate a variety of tasks -- primarily sentence- and discourse-level translation -- between ASL and 200 other languages as text, or 102 languages as speech. We provide baselines for tasks from ASL to English text using a unified modeling approach that incorporates timestamp tokens and previous text tokens in a 34-second context window, trained on random video clips from YouTube-ASL. This model meets or exceeds the performance of phrase-level baselines while supporting a multitude of new tasks. We also use FLEURS-ASL to show that multimodal frontier models have virtually no understanding of ASL, underscoring the importance of including sign languages in standard evaluation suites.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 30 of its 49 references that are in the map. First posted on arXiv 24 Aug 2024.

Builds on

30 of 49 references.

References · 30

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  5. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  6. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  7. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  8. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 583
  9. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  10. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  11. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  12. 2019Evaluating Gender Bias in Machine Translationcited by 249
18 more
  1. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  2. 2020How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Languagecited by 173
  3. 2021Learning Compact Metrics for MTcited by 93
  4. 2017Evaluating Discourse Phenomena in Neural Machine Translationcited by 87
  5. 2023YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpuscited by 75
  6. 2023A Benchmark for Learning to Translate a New Language from One Grammar Bookcited by 75
  7. 2022Open-Domain Sign Language Translation Learned from Online Videocited by 72
  8. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  9. 2021Including Signed Languages in Natural Language Processingcited by 68
  10. 2019The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-Englishcited by 62
  11. 2021SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognitioncited by 60
  12. 2024Towards Privacy-Aware Sign Language Translation at Scalecited by 44
  13. 2021Aligning Subtitles in Sign Language Videoscited by 26
  14. 2021Evaluating the Immediate Applicability of Pose Estimation for Sign Language Recognitioncited by 25
  15. 2020Evaluating Gender Bias in Speech Translationcited by 19
  16. 2024SignMusketeers: An Efficient Multi-Stream Approach for Sign Language Translation at Scalecited by 19
  17. 2023JWSign: A Highly Multilingual Corpus of Bible Translations for more Diversity in Sign Language Processingcited by 18
  18. 2023Is context all you need? Scaling Neural Sign Language Translation to Large Domains of Discoursecited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 6.

10 papers.

  1. 2025Multilingual Gloss-free Sign Language Translation: Towards Building a Sign Language Foundation Modelcited by 7
  2. 2025Lost in Translation, Found in Embeddings: Sign Language Translation and Alignmentcited by 6
  3. 20242M-BELEBELE: Highly Multilingual Speech and American Sign Language Comprehension Datasetcited by 3
  4. 2026Bootstrapping Sign Language Annotations with Sign Language Modelscited by 1
  5. 2025The TUB Sign Language Corpus Collectioncited by 0
  6. 2026Beyond a Single Reference: Training and Evaluation with Paraphrases in Sign Language Translationcited by 0
  7. 2026Targeted Linguistic Analysis of Sign Language Models with Minimal Translation Pairscited by 0
  8. 2026SignMatch: Matching Dictionary Signs to Continuous Sign Language Videocited by 0
  9. 2026SignDino: Self-Supervised Sign Language Representation Learning via Temporal-Axis Self-Distillationcited by 0
  10. 2026SignGPT: Toward LLM-Mediated Sign Language Interaction through Gloss-Free Translation and Generationcited by 0