mlsanityData to 14 Oct 2026

Leveraging the Power of MLLMs for Gloss-Free Sign Language Translation

arXiv 2411.16789 ↗First posted 25 Nov 2024cs.CV cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Sign language translation (SLT) is a challenging task that involves translating sign language images into spoken language. For SLT models to perform this task successfully, they must bridge the modality gap and identify subtle variations in sign language components to understand their meanings accurately. To address these challenges, we propose a novel gloss-free SLT framework called Multimodal Sign Language Translation (MMSLT), which leverages the representational capabilities of off-the-shelf multimodal large language models (MLLMs). Specifically, we use MLLMs to generate detailed textual descriptions of sign language components. Then, through our proposed multimodal-language pre-training module, we integrate these description features with sign video features to align them within the spoken sentence space. Our approach achieves state-of-the-art performance on benchmark datasets PHOENIX14T and CSL-Daily, highlighting the potential of MLLMs to be utilized effectively in SLT. Code is available at https://github.com/hwjeon98/MMSLT.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 43 of its 67 references that are in the map. First posted on arXiv 25 Nov 2024.

Builds on

43 of 67 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
31 more
  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  2. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  4. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  5. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  6. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
  7. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  8. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  9. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  10. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  11. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  12. 2020Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translationcited by 256
  13. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  14. 2023Multimodal Large Language Models: A Surveycited by 195
  15. 2021Improving Sign Language Translation with Monolingual Data by Sign Back-Translationcited by 179
  16. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  17. 2022Two-Stream Network for Sign Language Recognition and Translationcited by 123
  18. 2023Gloss-free Sign Language Translation: Improving from Visual-Language Pretrainingcited by 105
  19. 2022A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translationcited by 80
  20. 2024LLMs are Good Sign Language Translatorscited by 77
  21. 2020TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translationcited by 75
  22. 2024Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translationcited by 57
  23. 2023Gloss Attention for Gloss-free Sign Language Translationcited by 51
  24. 2021Stochastic Transformer Networks with Linear Competing Units: Application to end-to-end SL Translationcited by 35
  25. 2023SLTUNET: A Simple Unified Model for Sign Language Translationcited by 35
  26. 2024EgoLM: Multi-Modal Language Model of Egocentric Motionscited by 28
  27. 2021SimulSLT: End-to-End Simultaneous Sign Language Translationcited by 27
  28. 2024Factorized Learning Assisted with Large Language Model for Gloss-free Sign Language Translationcited by 24
  29. 2022A Token-level Contrastive Framework for Sign Language Translationcited by 10
  30. 2024An Efficient Sign Language Translation Using Spatial Configuration and Motion Dynamics with LLMscited by 9
  31. 2024MoChat: Joints-Grouped Spatio-Temporal Grounding LLM for Multi-Turn Motion Comprehension and Descriptioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 11.

13 papers.

Sort:

By area:

  1. 2026Gloss-Free Sign Language Translation: An Unbiased Evaluation of Progress in the Fieldcited by 9
  2. 2026MaDiS: Taming Masked Diffusion Language Models for Sign Language Generationcited by 4
  3. 2025Large Sign Language Models: Toward 3D American Sign Language Translationcited by 1
  4. 2025Bangla Sign Language Translation: Dataset Creation Challenges, Benchmarking and Prospectscited by 1
  5. 2026SignAgent: Agentic LLMs for Linguistically-Grounded Sign Language Annotation and Dataset Curationcited by 1
  6. 2026SIGNET: Motion-Level Knowledge Transfer for Cross-Language Sign Language Translationcited by 1
  7. 2026Towards Multi-View Sign Language Understanding: A Benchmark Dataset and Baselinecited by 0
  8. 2026Direct Translation between Sign Languagescited by 0
  9. 2026SLU-2K: A Question-Based Benchmark for Semantic Evaluation of Sign Language Translationcited by 0
  10. 2026ViPo-MLLM: Visual-Pose Multimodal LLM for Gloss-Free Sign Language Translationcited by 0
  11. 2026Q-BridgeNet: A Quantization Network for Cross-Lingual Sign Language Translationcited by 0
  12. 2026SMART: MLLM-guided Temporal Alignment for Unifying Sign Language Recognition and Spottingcited by 0
1 more
  1. 2026Beyond BLEU: A Case for Redefining Sign Language Translation Benchmarkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LLaVA-SLT: Visual Language Tuning for Sign Language Translation
  2. 2025Hierarchical Feature Alignment for Gloss-Free Sign Language Translation
  3. 2024DiffSLT: Enhancing Diversity in Sign Language Translation via Diffusion Model
  4. 2025Beyond Gloss: A Hand-Centric Framework for Gloss-Free Sign Language Translation
  5. 2023Cross-modality Data Augmentation for End-to-End Sign Language Translation
  6. 2023Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment
  7. 2025RVLF: A Reinforcing Vision-Language Framework for Gloss-Free Sign Language Translation
  8. 2024SignAttention: On the Interpretability of Transformer Models for Sign Language Translation