# SignLlama: Enhancing Gloss-free Sign Language Translation by Prioritizing Visual Features for LLMs

arXiv: https://arxiv.org/abs/2608.09006 · HTML: https://mlsanity.com/paper/2608.09006/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 42 references that are in the map.
- First posted on arXiv 10 Aug 2026.

## Builds on (most cited first, 16 of 16)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning](https://mlsanity.com/paper/2302.14115/) (2023)
- [Improving Sign Language Translation with Monolingual Data by Sign Back-Translation](https://mlsanity.com/paper/2105.12397/) (2021)
- [How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language](https://mlsanity.com/paper/2008.08143/) (2020)
- [Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning](https://mlsanity.com/paper/2402.11435/) (2024)
- [MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens](https://mlsanity.com/paper/2404.03413/) (2024)
- [VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding](https://mlsanity.com/paper/2406.09418/) (2024)
- [Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining](https://mlsanity.com/paper/2307.14768/) (2023)
- [LLMs are Good Sign Language Translators](https://mlsanity.com/paper/2404.00925/) (2024)
- [Towards Privacy-Aware Sign Language Translation at Scale](https://mlsanity.com/paper/2402.09611/) (2024)
- [Improving Gloss-free Sign Language Translation by Reducing Representation Density](https://mlsanity.com/paper/2405.14312/) (2024)
- [Is context all you need? Scaling Neural Sign Language Translation to Large Domains of Discourse](https://mlsanity.com/paper/2308.09622/) (2023)
- [LLaVA-SLT: Visual Language Tuning for Sign Language Translation](https://mlsanity.com/paper/2412.16524/) (2024)
- [Sign Language Question Answering: A New Task, Benchmark, and Baseline for Sign Language Understanding](https://mlsanity.com/paper/2607.27826/) (2026)

## Built on by (most cited first, 1 of 1)

- [OneSign: Unifying Sign Language Understanding Tasks with One Model](https://mlsanity.com/paper/2609.33090/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have achieved remarkable success across a wide range of tasks. However, fine-tuning LLMs for Gloss-Free Sign Language Translation (GFSLT) remains a challenge. In this paper, we investigate how to effectively adapt LLMs to the GFSLT task. We show that there are two key issues that need to be solved: (1) the inherent distributional gap between visual feature inputs and text feature inputs makes it difficult for LLMs to interpret visual inputs; and (2) existing approaches typically concatenate visual and textual features in an autoregressive framework, which leads to the model overemphasizing textual inputs and deprioritizing visual cues, as LLMs are pretrained predominantly on text-centric data. To address the first challenge, we propose a simple yet effective method named Filtered Pseudo-Gloss CTC Pretraining, which leverages filtered pseudo-gloss sequences generated from text sequences to supervise the training of the visual backbone. To tackle the second issue, we introduce a Visual-Prioritized Distillation training strategy. Specifically, we define a visual-only prediction path in which text inputs are masked, and the model is required to generate the target sequence relying solely on visual inputs. To guide this path, the outputs from the standard visual-textual prediction are then distilled into the visual-only prediction path, encouraging the model to prioritize visual features. Comprehensive experiments and qualitative analyses demonstrate the effectiveness of the proposed model. The proposed SignLlama achieves very competitive performance on multiple datasets for GFSLT tasks, without using any extra modalities or external sign language datasets for pretraining.