mlsanityData to 7 Oct 2026

Seeing, Signing, and Saying: A Vision-Language Model-Assisted Pipeline for Sign Language Data Acquisition and Curation from Social Media

arXiv 2510.25413 ↗First posted 29 Oct 2025cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Most existing sign language translation (SLT) datasets are limited in scale, lack multilingual coverage, and are costly to curate due to their reliance on expert annotation and controlled recording setup. Recently, Vision Language Models (VLMs) have demonstrated strong capabilities as evaluators and real-time assistants. Despite these advancements, their potential remains untapped in the context of sign language dataset acquisition. To bridge this gap, we introduce the first automated annotation and filtering framework that utilizes VLMs to reduce reliance on manual effort while preserving data quality. Our method is applied to TikTok videos across eight sign languages and to the already curated YouTube-SL-25 dataset in German Sign Language for the purpose of additional evaluation. Our VLM-based pipeline includes a face visibility detection, a sign activity recognition, a text extraction from video content, and a judgment step to validate alignment between video and text, implementing generic filtering, annotation and validation steps. Using the resulting corpus, TikTok-SL-8, we assess the performance of two off-the-shelf SLT models on our filtered dataset for German and American Sign Languages, with the goal of establishing baselines and evaluating the robustness of recent models on automatically extracted, slightly noisy data. Our work enables scalable, weakly supervised pretraining for SLT and facilitates data acquisition from social media.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 41 references that are in the map. First posted on arXiv 29 Oct 2025.

Builds on

29 of 41 references.

References · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  5. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  7. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  8. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  9. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  10. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  11. 2020Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translationcited by 254
  12. 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
17 more
  1. 2019Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparisoncited by 217
  2. 2021Improving Sign Language Translation with Monolingual Data by Sign Back-Translationcited by 177
  3. 2020How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Languagecited by 172
  4. 2020MLSUM: The Multilingual Summarization Corpuscited by 121
  5. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 119
  6. 2024LLMs are Good Sign Language Translatorscited by 76
  7. 2023YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpuscited by 74
  8. 2025Uni-Sign: Toward Unified Sign Language Understanding at Scalecited by 55
  9. 2024Towards Privacy-Aware Sign Language Translation at Scalecited by 42
  10. 2023Sign Language Translation from Instructional Videoscited by 39
  11. 2024YouTube-SL-25: A Large-Scale, Open-Domain Multilingual Sign Language Parallel Corpuscited by 36
  12. 2021Content4All Open Research Sign Language Translation Datasetscited by 30
  13. 2023JWSign: A Highly Multilingual Corpus of Bible Translations for more Diversity in Sign Language Processingcited by 18
  14. 2024CoCoLoFa: A Dataset of News Comments with Common Logical Fallacies Written by LLM-Assisted Crowdscited by 8
  15. 2025Sign Language Translation with Sentence Embedding Supervisioncited by 6
  16. 2024VolDoGer: LLM-assisted Datasets for Domain Generalization in Vision-Language Taskscited by 3
  17. 2024Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Sign Language Recognition in the Age of LLMs
  2. 2026SignNet-1M: Large-Scale Multilingual Sign Language Video Dataset with Downstream Benchmarks
  3. 2026Bootstrapping Sign Language Annotations with Sign Language Models
  4. 2024LLaVA-SLT: Visual Language Tuning for Sign Language Translation
  5. 2024Leveraging the Power of MLLMs for Gloss-Free Sign Language Translation
  6. 2026Corpus Augmentation for Sign Language Translation via LLM-Guided Video Stitching
  7. 2026SLU-2K: A Question-Based Benchmark for Semantic Evaluation of Sign Language Translation
  8. 2026Attention-Steered Vision-Language Models for Sign Language Translation