mlsanityData to 14 Oct 2026

VTaMo: Video-Text Alignment Model for Sign Language Translation

arXiv 2607.09126 ↗First posted 10 Jul 2026cs.CV cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Sign language translation (SLT) converts continuous sign videos into spoken language text. Gloss-free approaches leverage pre-trained visual encoders and language models but rely on implicit cross-modal alignment from translation supervision alone. We present VTaMo, a framework that introduces explicit multi-granularity alignment at three levels: (1) local alignment via entropy-regularized optimal transport with a learnable null token for fine-grained frame-to-token correspondences; (2) global alignment via a learnable orthogonal transformation that calibrates embedding space geometry through Earth Mover's Distance; and (3) position-aligned contrastive learning for discriminative token-level representations. Experiments on Phoenix-2014T, CSL-Daily, How2Sign, and OpenASL demonstrate consistent state-of-the-art performance, with ablations confirming the complementary contributions of each component. Code is available at https://github.com/junyi2005/vtamo.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 36 of its 47 references that are in the map. First posted on arXiv 10 Jul 2026.

Builds on

36 of 47 references.

References · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  8. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  9. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  10. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  11. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  12. 2017Word Translation Without Parallel Datacited by 487
24 more
  1. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  2. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 280
  3. 2020Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translationcited by 256
  4. 2021Improving Sign Language Translation with Monolingual Data by Sign Back-Translationcited by 179
  5. 2020How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Languagecited by 173
  6. 2022Two-Stream Network for Sign Language Recognition and Translationcited by 123
  7. 2023Gloss-free Sign Language Translation: Improving from Visual-Language Pretrainingcited by 105
  8. 2022A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translationcited by 80
  9. 2024LLMs are Good Sign Language Translatorscited by 77
  10. 2020TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translationcited by 75
  11. 2023YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpuscited by 75
  12. 2022Open-Domain Sign Language Translation Learned from Online Videocited by 72
  13. 2024When Do We Not Need Larger Vision Models?cited by 67
  14. 2024Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translationcited by 57
  15. 2025Uni-Sign: Toward Unified Sign Language Understanding at Scalecited by 56
  16. 2023Gloss Attention for Gloss-free Sign Language Translationcited by 51
  17. 2024Towards Privacy-Aware Sign Language Translation at Scalecited by 44
  18. 2023SLTUNET: A Simple Unified Model for Sign Language Translationcited by 35
  19. 2023Gloss-Free End-to-End Sign Language Translationcited by 32
  20. 2023Learning to Ground Instructional Articles in Videos through Narrationscited by 28
  21. 2024C2RL: Content and Context Representation Learning for Gloss-free Sign Language Translation and Retrievalcited by 24
  22. 2024SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Predictioncited by 22
  23. 2022A Token-level Contrastive Framework for Sign Language Translationcited by 10
  24. 2023VK-G2T: Vision and Context Knowledge enhanced Gloss2Textcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Hierarchical Feature Alignment for Gloss-Free Sign Language Translation
  2. 2026Corpus Augmentation for Sign Language Translation via LLM-Guided Video Stitching
  3. 2023Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment
  4. 2025Beyond Gloss: A Hand-Centric Framework for Gloss-Free Sign Language Translation
  5. 2026Selective Contrastive Learning For Gloss Free Sign Language Translation
  6. 2023Cross-modality Data Augmentation for End-to-End Sign Language Translation
  7. 2026SignFLIP: A Unified Model for Sign Language Translation and Generation via Stage-wise Alignment at Scale
  8. 2024LLaVA-SLT: Visual Language Tuning for Sign Language Translation