mlsanityData to 7 Oct 2026

LLMs are Good Sign Language Translators

arXiv 2404.00925 ↗First posted 1 Apr 2024cs.CV cs.CLcited by 76

Not on the map. Why?

Abstract (arXiv)

Sign Language Translation (SLT) is a challenging task that aims to translate sign videos into spoken language. Inspired by the strong translation capabilities of large language models (LLMs) that are trained on extensive multilingual text corpora, we aim to harness off-the-shelf LLMs to handle SLT. In this paper, we regularize the sign videos to embody linguistic characteristics of spoken language, and propose a novel SignLLM framework to transform sign videos into a language-like representation for improved readability by off-the-shelf LLMs. SignLLM comprises two key modules: (1) The Vector-Quantized Visual Sign module converts sign videos into a sequence of discrete character-level sign tokens, and (2) the Codebook Reconstruction and Alignment module converts these character-level tokens into word-level sign representations using an optimal transport formulation. A sign-text alignment loss further bridges the gap between sign and text tokens, enhancing semantic compatibility. We achieve state-of-the-art gloss-free results on two widely-used SLT benchmarks.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Builds on 48 of its 74 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

48 of 74 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  11. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  12. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
36 more
  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  4. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,417
  5. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  6. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  7. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  8. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  9. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  10. 2020Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translationcited by 254
  11. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  12. 2021Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answeringcited by 178
  13. 2021Improving Sign Language Translation with Monolingual Data by Sign Back-Translationcited by 177
  14. 2020How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Languagecited by 172
  15. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 169
  16. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  17. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  18. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  19. 2022Two-Stream Network for Sign Language Recognition and Translationcited by 122
  20. 2023Gloss-free Sign Language Translation: Improving from Visual-Language Pretrainingcited by 104
  21. 2021Survey of Low-Resource Machine Translationcited by 92
  22. 2020Fast WordPiece Tokenizationcited by 85
  23. 2022A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translationcited by 80
  24. 2020TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translationcited by 75
  25. 2018Neural Sign Language Translation based on Human Keypoint Estimationcited by 65
  26. 2019Revisiting Low-Resource Neural Machine Translation: A Case Studycited by 61
  27. 2020Vocabulary Learning via Optimal Transport for Neural Machine Translationcited by 58
  28. 2023Gloss Attention for Gloss-free Sign Language Translationcited by 50
  29. 2024LLMs are Good Action Recognizerscited by 36
  30. 2021Stochastic Transformer Networks with Linear Competing Units: Application to end-to-end SL Translationcited by 35
  31. 2023SLTUNET: A Simple Unified Model for Sign Language Translationcited by 35
  32. 2019A Call for Prudent Choice of Subword Merge Operations in Neural Machine Translationcited by 30
  33. 2021Sign Language Translation with Hierarchical Spatio-TemporalGraph Neural Networkcited by 26
  34. 2023AI-Generated Content (AIGC) for Various Data Modalities: A Surveycited by 25
  35. 2022Reprogramming Pretrained Language Models for Antibody Sequence Infillingcited by 20
  36. 2022A Token-level Contrastive Framework for Sign Language Translationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 29; 2026: 30.

76 papers.

Sort:

By area:

  1. 2025Uni-Sign: Toward Unified Sign Language Understanding at Scalecited by 55
  2. 2025Lost in Translation, Found in Context: Sign Language Translation with Contextual Cuescited by 25
  3. 2024C2RL: Content and Context Representation Learning for Gloss-free Sign Language Translation and Retrievalcited by 23
  4. 2024SignCLIP: Connecting Text and Sign Language by Contrastive Learningcited by 20
  5. 2024Scaling up Multimodal Pre-training for Sign Language Understandingcited by 20
  6. 2024Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generatorcited by 20
  7. 2025Jailbreak-AudioBench: In-Depth Evaluation and Analysis of Jailbreak Threats for Large Audio Language Modelscited by 17
  8. 2024Discrete to Continuous: Generating Smooth Transition Poses from Sign Language Observationcited by 16
  9. 2024Leveraging the Power of MLLMs for Gloss-Free Sign Language Translationcited by 13
  10. 2025Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translationcited by 12
  11. 2025Geo-Sign: Hyperbolic Contrastive Regularisation for Geometrically Aware Sign Language Translationcited by 11
  12. 2024Speculative Coreset Selection for Task-Specific Fine-tuningcited by 10
38 more
  1. 2024An Efficient Sign Language Translation Using Spatial Configuration and Motion Dynamics with LLMscited by 9
  2. 2026Gloss-Free Sign Language Translation: An Unbiased Evaluation of Progress in the Fieldcited by 9
  3. 2025Multilingual Gloss-free Sign Language Translation: Towards Building a Sign Language Foundation Modelcited by 7
  4. 2025Beyond Gloss: A Hand-Centric Framework for Gloss-Free Sign Language Translationcited by 7
  5. 2025Isharah: A Large-Scale Multi-Scene Dataset for Continuous Sign Language Recognitioncited by 6
  6. 2024A Spatio-Temporal Representation Learning as an Alternative to Traditional Glosses in Sign Language Translation and Productioncited by 5
  7. 2024SceneLLM: Implicit Language Reasoning in LLM for Dynamic Scene Graph Generationcited by 5
  8. 2024SCOPE: Sign Language Contextual Processing with Embedding from LLMscited by 4
  9. 2025Sign Language Translation using Frame and Event Stream: Benchmark Dataset and Algorithmscited by 4
  10. 2024SituationalLLM: Proactive language models with scene awareness for dynamic, contextual task guidancecited by 3
  11. 2024Modelling the Distribution of Human Motion for Sign Language Assessmentcited by 3
  12. 2025Text-Driven Diffusion Model for Sign Language Productioncited by 3
  13. 2025Grounding or Guessing? Visual Signals for Detecting Hallucinations in Sign Language Translationcited by 3
  14. 2025SONAR-SLT: Multilingual Sign Language Translation via Language-Agnostic Sentence Embedding Supervisioncited by 3
  15. 2026MaDiS: Taming Masked Diffusion Language Models for Sign Language Generationcited by 3
  16. 2025RVLF: A Reinforcing Vision-Language Framework for Gloss-Free Sign Language Translationcited by 2
  17. 2026CNSL-bench: Benchmarking the Sign Language Understanding Capabilities of MLLMs on Chinese National Sign Languagecited by 2
  18. 2024ELMI: Interactive and Intelligent Sign Language Translation of Lyrics for Song Signingcited by 1
  19. 2024Diverse Sign Language Translationcited by 1
  20. 2024Enhanced Sign Language Translation between American Sign Language (ASL) and Indian Sign Language (ISL) Using LLMscited by 1
  21. 2024DiffSLT: Enhancing Diversity in Sign Language Translation via Diffusion Modelcited by 1
  22. 2025SAGE: Segment-Aware Gloss-Free Encoding for Token-Efficient Sign Language Translationcited by 1
  23. 2025Contrastive Pretraining with Dual Visual Encoders for Gloss-Free Sign Language Translationcited by 1
  24. 2025Motion is the Choreographer: Learning Latent Pose Dynamics for Seamless Sign Language Generationcited by 1
  25. 2025Leveraging Large Language Models for Accurate Sign Language Translation in Low-Resource Scenarioscited by 1
  26. 2025Towards Inclusive Communication: A Unified Framework for Generating Spoken Language from Sign, Lip, and Audiocited by 1
  27. 2025SignBind-LLM: Multi-Stage Modality Fusion for Sign Language Translationcited by 1
  28. 2025Text2Sign Diffusion: A Generative Approach for Gloss-Free Sign Language Productioncited by 1
  29. 2025Saudi Sign Language Translation Using T5cited by 1
  30. 2025Large Sign Language Models: Toward 3D American Sign Language Translationcited by 1
  31. 2026SignAgent: Agentic LLMs for Linguistically-Grounded Sign Language Annotation and Dataset Curationcited by 1
  32. 2026Think in Latent Thoughts: A New Paradigm for Gloss-Free Sign Language Translationcited by 1
  33. 2026Mol-Debate: Multi-Agent Debate Improves Structural Reasoning in Molecular Designcited by 1
  34. 2026Context-Aware Autoregressive Diffusion for Gloss-Wise Sign Language Productioncited by 1
  35. 2026SIGNET: Motion-Level Knowledge Transfer for Cross-Language Sign Language Translationcited by 1
  36. 2026BackTranslation2.0 -- A Linguistically Motivated Metric to Assess Sign Language Productioncited by 1
  37. 2026SignLlama: Enhancing Gloss-free Sign Language Translation by Prioritizing Visual Features for LLMscited by 1
  38. 2025Beyond Words: AuralLLM and SignMST-C for Sign Language Production and Bidirectional Accessibilitycited by 0

50 most cited of 76 All 76 (JSON)