mlsanityData to 7 Oct 2026

Gloss Attention for Gloss-free Sign Language Translation

arXiv 2307.07361 ↗First posted 14 Jul 2023cs.CV cs.CLcited by 50

Not on the map. Why?

Abstract (arXiv)

Most sign language translation (SLT) methods to date require the use of gloss annotations to provide additional supervision information, however, the acquisition of gloss is not easy. To solve this problem, we first perform an analysis of existing models to confirm how gloss annotations make SLT easier. We find that it can provide two aspects of information for the model, 1) it can help the model implicitly learn the location of semantic boundaries in continuous sign language videos, 2) it can help the model understand the sign language video globally. We then propose gloss attention, which enables the model to keep its attention within video segments that have the same semantics locally, just as gloss helps existing models do. Furthermore, we transfer the knowledge of sentence-to-sentence similarity from the natural language model to our gloss attention SLT network (GASLT) to help it understand sign language videos at the sentence level. Experimental results on multiple large-scale sign language datasets show that our proposed GASLT model significantly outperforms existing methods. Our code is provided in urlhttps://github.com/YinAoXiong/GASLT.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Builds on 44 of its 79 references that are in the map. First posted on arXiv 14 Jul 2023.

Builds on

44 of 79 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  10. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
32 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  2. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  3. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  4. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  5. 2014Distributed Representations of Sentences and Documentscited by 677
  6. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  7. 2015Skip-Thought Vectorscited by 321
  8. 2020Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translationcited by 254
  9. 2019Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparisoncited by 217
  10. 2021Improving Sign Language Translation with Monolingual Data by Sign Back-Translationcited by 177
  11. 2022ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speechcited by 147
  12. 2022Two-Stream Network for Sign Language Recognition and Translationcited by 122
  13. 2018MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Languagecited by 118
  14. 2018Video-based Sign Language Recognition without Temporal Segmentationcited by 88
  15. 2022A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translationcited by 80
  16. 2021Visual Alignment Constraint for Continuous Sign Language Recognitioncited by 79
  17. 2020TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translationcited by 75
  18. 2020Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognitioncited by 74
  19. 2020Multi-channel Transformers for Multi-articulatory Sign Language Translationcited by 71
  20. 2018Neural Sign Language Translation based on Human Keypoint Estimationcited by 65
  21. 2020Transferring Cross-domain Knowledge for Video Sign Language Recognitioncited by 60
  22. 2020Fully Convolutional Networks for Continuous Sign Language Recognitioncited by 58
  23. 2017BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languagescited by 52
  24. 2019Joey NMT: A Minimalist NMT Toolkit for Novicescited by 39
  25. 2021Stochastic Transformer Networks with Linear Competing Units: Application to end-to-end SL Translationcited by 35
  26. 2022TranSpeech: Speech-to-Speech Translation With Bilateral Perturbationcited by 34
  27. 2021SimulSLT: End-to-End Simultaneous Sign Language Translationcited by 26
  28. 2018Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representationscited by 20
  29. 2020Neural Sign Language Translation by Learning Tokenizationcited by 19
  30. 2021SimulLR: Simultaneous Lip Reading Transducer with Attention-Guided Adaptive Memorycited by 7
  31. 2022Video-Guided Curriculum Learning for Spoken Video Groundingcited by 5
  32. 2023DATE: Domain Adaptive Product Seeker for E-commercecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 15; 2025: 20; 2026: 12.

50 papers.

Sort:

By area:

  1. 2023Gloss-free Sign Language Translation: Improving from Visual-Language Pretrainingcited by 104
  2. 2024LLMs are Good Sign Language Translatorscited by 76
  3. 2024Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translationcited by 56
  4. 2025Uni-Sign: Toward Unified Sign Language Understanding at Scalecited by 55
  5. 2024Improving Gloss-free Sign Language Translation by Reducing Representation Densitycited by 30
  6. 2024C2RL: Content and Context Representation Learning for Gloss-free Sign Language Translation and Retrievalcited by 23
  7. 2024Scaling up Multimodal Pre-training for Sign Language Understandingcited by 20
  8. 2024A Tale of Two Languages: Large-Vocabulary Continuous Sign Language Recognition from Spoken Language Supervisioncited by 15
  9. 2024Leveraging the Power of MLLMs for Gloss-Free Sign Language Translationcited by 13
  10. 2025Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translationcited by 12
  11. 2025Geo-Sign: Hyperbolic Contrastive Regularisation for Geometrically Aware Sign Language Translationcited by 11
  12. 2023Autoregressive Sign Language Production: A Gloss-Free Approach with Discrete Representationscited by 9
38 more
  1. 2026Gloss-Free Sign Language Translation: An Unbiased Evaluation of Progress in the Fieldcited by 9
  2. 2025Beyond Gloss: A Hand-Centric Framework for Gloss-Free Sign Language Translationcited by 7
  3. 2024EvSign: Sign Language Recognition and Translation with Streaming Eventscited by 6
  4. 2025Isharah: A Large-Scale Multi-Scene Dataset for Continuous Sign Language Recognitioncited by 6
  5. 2025Hierarchical Feature Alignment for Gloss-Free Sign Language Translationcited by 6
  6. 2024A Spatio-Temporal Representation Learning as an Alternative to Traditional Glosses in Sign Language Translation and Productioncited by 5
  7. 2025Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context Learningcited by 5
  8. 2025Sign Language Translation using Frame and Event Stream: Benchmark Dataset and Algorithmscited by 4
  9. 2024Modelling the Distribution of Human Motion for Sign Language Assessmentcited by 3
  10. 2025Understanding Co-speech Gestures in-the-wildcited by 3
  11. 2023Language Model is a Branch Predictor for Simultaneous Machine Translationcited by 2
  12. 2024A Chinese Continuous Sign Language Dataset Based on Complex Environmentscited by 2
  13. 2025Exploring Pose-based Sign Language Translation: Ablation Studies and Attention Insightscited by 2
  14. 2025RVLF: A Reinforcing Vision-Language Framework for Gloss-Free Sign Language Translationcited by 2
  15. 2024Diverse Sign Language Translationcited by 1
  16. 2024DiffSLT: Enhancing Diversity in Sign Language Translation via Diffusion Modelcited by 1
  17. 2025ADAT: Time-Series-Aware Adaptive Transformer Architecture for Sign Language Translationcited by 1
  18. 2025SAGE: Segment-Aware Gloss-Free Encoding for Token-Efficient Sign Language Translationcited by 1
  19. 2025SignBind-LLM: Multi-Stage Modality Fusion for Sign Language Translationcited by 1
  20. 2025Text2Sign Diffusion: A Generative Approach for Gloss-Free Sign Language Productioncited by 1
  21. 2025Saudi Sign Language Translation Using T5cited by 1
  22. 2025Bangla Sign Language Translation: Dataset Creation Challenges, Benchmarking and Prospectscited by 1
  23. 2026BackTranslation2.0 -- A Linguistically Motivated Metric to Assess Sign Language Productioncited by 1
  24. 2024Event Stream-based Sign Language Translation: A High-Definition Benchmark Dataset and A Novel Baselinecited by 0
  25. 2024SignAttention: On the Interpretability of Transformer Models for Sign Language Translationcited by 0
  26. 2025Graph Network for Sign Language Taskscited by 0
  27. 2025iLSU-T: an Open Dataset for Uruguayan Sign Language Translationcited by 0
  28. 2025SSL-SSAW: Self-Supervised Learning with Sigmoid Self-Attention Weighting for Question-Based Sign Language Translationcited by 0
  29. 2026A2V-SLP: Alignment-Aware Variational Modeling for Disentangled Sign Language Productioncited by 0
  30. 2026AI-Driven Modular Services for Accessible Multilingual Education in Immersive Extended Reality Settings: Integrating Speech Processing, Translation, and Sign Language Renderingcited by 0
  31. 2026INTERACT: An AI-Driven Extended Reality Framework for Accesible Communication Featuring Real-Time Sign Language Interpretation and Emotion Recognitioncited by 0
  32. 2026SignDPO: Multi-level Direct Preference Optimisation for Skeleton-based Gloss-free Sign Language Translationcited by 0
  33. 2026Towards Multi-View Sign Language Understanding: A Benchmark Dataset and Baselinecited by 0
  34. 2026SignNet-1M: Large-Scale Multilingual Sign Language Video Dataset with Downstream Benchmarkscited by 0
  35. 2026ViPo-MLLM: Visual-Pose Multimodal LLM for Gloss-Free Sign Language Translationcited by 0
  36. 2026VTaMo: Video-Text Alignment Model for Sign Language Translationcited by 0
  37. 2026ODE-Based Transformer Decoders for Iterative Sign Language Translationcited by 0
  38. 2026OneSign: Unifying Sign Language Understanding Tasks with One Modelcited by 0