mlsanityData to 14 Oct 2026

ViPo-MLLM: Visual-Pose Multimodal LLM for Gloss-Free Sign Language Translation

arXiv 2607.03657 ↗First posted 4 Jul 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Gloss-free Sign Language Translation (SLT) translates sign language videos into spoken-language sentences without gloss annotations, avoiding costly labeling but requiring fine-grained modeling of hands, body, and facial cues. Existing methods often use single-modality or weakly fused features, limiting performance. We propose ViPo-MLLM, a framework that integrates spatio-temporal RGB and human pose features. Dedicated encoders model intra-modal dynamics and cross-modal attention captures long-range dependencies. The fused representation is conditioned with a structured prompt and processed by an LLM trained with contrastive and language modeling objectives. The proposed model was evaluated on the PHOENIX14T and CSL-Daily datasets and achieved new state-of-the-art results on both datasets. Moreover, the ViPo-MLLM model attained competitive performance compared to gloss-based recognition approaches, confirming the effectiveness of the proposed pose cues and cross-modal attention mechanisms.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 35 references that are in the map. First posted on arXiv 4 Jul 2026.

Builds on

24 of 35 references.

References · 24

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  3. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,162
  4. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  5. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 508
  6. 2020Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translationcited by 256
  7. 2021Improving Sign Language Translation with Monolingual Data by Sign Back-Translationcited by 179
  8. 2022Two-Stream Network for Sign Language Recognition and Translationcited by 123
  9. 2023Gloss-free Sign Language Translation: Improving from Visual-Language Pretrainingcited by 105
  10. 2024LLMs are Good Sign Language Translatorscited by 77
  11. 2020TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translationcited by 75
  12. 2020Multi-channel Transformers for Multi-articulatory Sign Language Translationcited by 71
12 more
  1. 2024Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translationcited by 57
  2. 2023Gloss Attention for Gloss-free Sign Language Translationcited by 51
  3. 2021Stochastic Transformer Networks with Linear Competing Units: Application to end-to-end SL Translationcited by 35
  4. 2025Lost in Translation, Found in Context: Sign Language Translation with Contextual Cuescited by 26
  5. 2024Factorized Learning Assisted with Large Language Model for Gloss-free Sign Language Translationcited by 24
  6. 2023Cross-modality Data Augmentation for End-to-End Sign Language Translationcited by 16
  7. 2024Leveraging the Power of MLLMs for Gloss-Free Sign Language Translationcited by 13
  8. 2022A Token-level Contrastive Framework for Sign Language Translationcited by 10
  9. 2024An Efficient Sign Language Translation Using Spatial Configuration and Motion Dynamics with LLMscited by 9
  10. 2025Lost in Translation, Found in Embeddings: Sign Language Translation and Alignmentcited by 6
  11. 2025AraReasoner: Evaluating Reasoning-Based LLMs for Arabic NLPcited by 5
  12. 2025USTM: Unified Spatial and Temporal Modeling for Continuous Sign Language Recognitioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Gloss: A Hand-Centric Framework for Gloss-Free Sign Language Translation
  2. 2024LLaVA-SLT: Visual Language Tuning for Sign Language Translation
  3. 2025RVLF: A Reinforcing Vision-Language Framework for Gloss-Free Sign Language Translation
  4. 2026Attention-Steered Vision-Language Models for Sign Language Translation
  5. 2026Corpus Augmentation for Sign Language Translation via LLM-Guided Video Stitching
  6. 2024DiffSLT: Enhancing Diversity in Sign Language Translation via Diffusion Model
  7. 2023Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment
  8. 2025Seeing, Signing, and Saying: A Vision-Language Model-Assisted Pipeline for Sign Language Data Acquisition and Curation from Social Media