mlsanityData to 14 Oct 2026

Attention-Steered Vision-Language Models for Sign Language Translation

arXiv 2608.00235 ↗First posted 31 Jul 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models (VLMs) have emerged as a powerful framework for multimodal video understanding. However, they remain limited in the sign language translation task, where we identify a key failure mode of existing VLMbased translators: poor spatial-temporal visual grounding. In particular, we find that standard next-token cross-entropy does not directly provide signal for where and when the model should attend, causing models to overlook sign-relevant regions and frames. To address this challenge, we propose AttnSign, a VLM-based spatial-temporal attention steering framework for sign language translation. AttnSign first introduces spatial attention supervision for sign-relevant regions, such as face and hands, in each frame; then develops an RL-based motion-cadence steering method that encourages the model to explore and focus on sign-level keyframes. Experimental results on How2Sign and OpenASL benchmarks show that our proposed AttnSign consistently outperforms existing methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 50 references that are in the map. First posted on arXiv 31 Jul 2026.

Builds on

33 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training4 Mar
  2. 2023Voila-A: Aligning Vision-Language Models with User's Gaze Attention22 Dec

Other references · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  5. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  6. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  7. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 583
  8. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 484
  9. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 461
  10. 2020Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translationcited by 256
  11. 2020How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Languagecited by 173
  12. 2019Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Groundedcited by 128
19 more
  1. 2023Gloss-free Sign Language Translation: Improving from Visual-Language Pretrainingcited by 105
  2. 2024LLMs are Good Sign Language Translatorscited by 77
  3. 2023YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpuscited by 75
  4. 2022Open-Domain Sign Language Translation Learned from Online Videocited by 72
  5. 2023SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understandingcited by 59
  6. 2024Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translationcited by 57
  7. 2025Uni-Sign: Toward Unified Sign Language Understanding at Scalecited by 56
  8. 2024Towards Privacy-Aware Sign Language Translation at Scalecited by 44
  9. 2023Sign Language Translation from Instructional Videoscited by 39
  10. 2025Lost in Translation, Found in Context: Sign Language Translation with Contextual Cuescited by 26
  11. 2024C2RL: Content and Context Representation Learning for Gloss-free Sign Language Translation and Retrievalcited by 24
  12. 2024SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Predictioncited by 22
  13. 2024SignMusketeers: An Efficient Multi-Stream Approach for Sign Language Translation at Scalecited by 19
  14. 2024LLaVA-SLT: Visual Language Tuning for Sign Language Translationcited by 14
  15. 2024An Efficient Sign Language Translation Using Spatial Configuration and Motion Dynamics with LLMscited by 9
  16. 2025Sign Language Translation with Sentence Embedding Supervisioncited by 6
  17. 2026Semantic Communities and Boundary-Spanning Lyrics in K-pop: A Graph-Based Unsupervised Analysiscited by 2
  18. 2026CNSL-bench: Benchmarking the Sign Language Understanding Capabilities of MLLMs on Chinese National Sign Languagecited by 2
  19. 2026Think in Latent Thoughts: A New Paradigm for Gloss-Free Sign Language Translationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Multi-Stream Keypoint Attention Network for Sign Language Recognition and Translation
  2. 2025Beyond Gloss: A Hand-Centric Framework for Gloss-Free Sign Language Translation
  3. 2024SignAttention: On the Interpretability of Transformer Models for Sign Language Translation
  4. 2025RVLF: A Reinforcing Vision-Language Framework for Gloss-Free Sign Language Translation
  5. 2026SignNet-1M: Large-Scale Multilingual Sign Language Video Dataset with Downstream Benchmarks
  6. 2026ViPo-MLLM: Visual-Pose Multimodal LLM for Gloss-Free Sign Language Translation
  7. 2024DiffSLT: Enhancing Diversity in Sign Language Translation via Diffusion Model
  8. 2026SignRefine: Adapting Foundational Video Models for Sign Language Generation