mlsanityData to 7 Oct 2026

Read and Attend: Temporal Localisation in Sign Language Videos

arXiv 2103.16481 ↗First posted 30 Mar 2021cs.CVcited by 34

Not on the map. Why?

Abstract (arXiv)

The objective of this work is to annotate sign instances across a broad vocabulary in continuous sign language. We train a Transformer model to ingest a continuous signing stream and output a sequence of written tokens on a large-scale collection of signing footage with weakly-aligned subtitles. We show that through this training it acquires the ability to attend to a large vocabulary of sign instances in the input sequence, enabling their localisation. Our contributions are as follows: (1) we demonstrate the ability to leverage large quantities of continuous signing videos with weakly-aligned subtitles to localise signs in continuous sign language; (2) we employ the learned attention to automatically generate hundreds of thousands of annotations for a large sign vocabulary; (3) we collect a set of 37K manually verified sign instances across a vocabulary of 950 sign classes to support our study of sign language recognition; (4) by training on the newly annotated data from our method, we outperform the prior state of the art on the BSL-1K sign language recognition benchmark.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 29 of its 55 references that are in the map. First posted on arXiv 30 Mar 2021.

Builds on

29 of 55 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  3. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  5. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  6. 2017Objects that Soundcited by 254
  7. 2020Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translationcited by 254
  8. 2019Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparisoncited by 217
  9. 2018Learning to Localize Sound Source in Visual Scenescited by 188
  10. 2019The VIA Annotation Software for Images, Audio and Videocited by 166
  11. 2018To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regressioncited by 157
  12. 2018Multilevel Language and Vision Integration for Text-to-Clip Retrievalcited by 128
17 more
  1. 2018MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Languagecited by 118
  2. 2020BSL-1K: Scaling up co-articulated sign language recognition using mouthing cuescited by 106
  3. 2018Video-based Sign Language Recognition without Temporal Segmentationcited by 88
  4. 2019Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspectivecited by 87
  5. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  6. 2019Jointly Learning to Align and Translate with Transformer Modelscited by 75
  7. 2020Multi-channel Transformers for Multi-articulatory Sign Language Translationcited by 71
  8. 2017Combining Residual Networks with LSTMs for Lipreadingcited by 66
  9. 2018Neural Sign Language Translation based on Human Keypoint Estimationcited by 65
  10. 2020Transferring Cross-domain Knowledge for Video Sign Language Recognitioncited by 60
  11. 2020Fully Convolutional Networks for Continuous Sign Language Recognitioncited by 58
  12. 2020Watch, read and lookup: learning to spot signs from multiple supervisorscited by 35
  13. 2020Seeing wake words: Audio-visual Keyword Spottingcited by 22
  14. 2018Efficient keyword spotting using dilated convolutions and gatingcited by 21
  15. 2018Attention-based End-to-End Models for Small-Footprint Keyword Spottingcited by 18
  16. 2018Zero-shot keyword spotting for visual speech recognition in-the-wildcited by 15
  17. 2016Temporal Action Localization in Untrimmed Videos via Multi-stage CNNscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 7; 2023: 5; 2024: 7; 2025: 6; 2026: 4.

34 papers.

Sort:

  1. 2022Multimodal Learning with Transformers: A Surveycited by 399
  2. 2022Open-Domain Sign Language Translation Learned from Online Videocited by 72
  3. 2021Sub-word Level Lip Reading With Visual Attentioncited by 60
  4. 2023CiCo: Domain-Aware Sign Language Retrieval via Cross-Lingual Contrastive Learningcited by 32
  5. 2021BBC-Oxford British Sign Language Datasetcited by 27
  6. 2021Skeletor: Skeletal Transformers for Robust Body-Pose Estimationcited by 26
  7. 2021Aligning Subtitles in Sign Language Videoscited by 26
  8. 2022Sign Language Video Retrieval with Free-Form Textual Queriescited by 21
  9. 2022Automatic dense annotation of large-vocabulary sign language videoscited by 20
  10. 2024SignCLIP: Connecting Text and Sign Language by Contrastive Learningcited by 20
  11. 2024Systemic Biases in Sign Language AI Research: A Deaf-Led Call to Reevaluate Research Agendascited by 19
  12. 2023JWSign: A Highly Multilingual Corpus of Bible Translations for more Diversity in Sign Language Processingcited by 18
22 more
  1. 2024A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatarscited by 18
  2. 2024SignMusketeers: An Efficient Multi-Stream Approach for Sign Language Translation at Scalecited by 18
  3. 2024A Tale of Two Languages: Large-Vocabulary Continuous Sign Language Recognition from Spoken Language Supervisioncited by 15
  4. 2024Towards Online Continuous Sign Language Recognition and Translationcited by 13
  5. 2022Scaling up sign spotting through sign language dictionariescited by 12
  6. 2021Visual Keyword Spotting with Attentioncited by 11
  7. 2025Geo-Sign: Hyperbolic Contrastive Regularisation for Geometrically Aware Sign Language Translationcited by 11
  8. 2022Weakly-supervised Fingerspelling Recognition in British Sign Language Videoscited by 10
  9. 2023Self-Supervised Video Transformers for Isolated Sign Language Recognitioncited by 6
  10. 2025Deep Understanding of Sign Language for Sign to Subtitle Alignmentcited by 6
  11. 2023Towards the extraction of robust sign embeddings for low resource sign language recognitioncited by 5
  12. 2025Text-Driven 3D Hand Motion Generation from Sign Language Datacited by 5
  13. 2025SiLVERScore: Semantically-Aware Embeddings for Sign Language Generation Evaluationcited by 4
  14. 2022Clean Text and Full-Body Transformer: Microsoft's Submission to the WMT22 Shared Task on Sign Language Translationcited by 3
  15. 2023Gloss Alignment Using Word Embeddingscited by 3
  16. 2025How Pragmatics Shape Articulation: A Computational Case Study in STEM ASL Discoursecited by 1
  17. 2024Transformer with Controlled Attention for Synchronous Motion Captioningcited by 0
  18. 2025iLSU-T: an Open Dataset for Uruguayan Sign Language Translationcited by 0
  19. 2026Towards Continuous Sign Language Conversation from Isolated Signscited by 0
  20. 2026Semantic Hardness Is Not Visual Hardness: Sign-Aware Hard Negative Mining for Sign Language Retrievalcited by 0
  21. 2026SMART: MLLM-guided Temporal Alignment for Unifying Sign Language Recognition and Spottingcited by 0
  22. 2026Seeing Semantic Shift: Difference-Aware Sentence-Level Temporal Segmentation of Sign Language Videoscited by 0