mlsanityData to 7 Oct 2026

Self-Supervised Video Transformers for Isolated Sign Language Recognition

arXiv 2309.02450 ↗First posted 2 Sep 2023cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

This paper presents an in-depth analysis of various self-supervision methods for isolated sign language recognition (ISLR). We consider four recently introduced transformer-based approaches to self-supervised learning from videos, and four pre-training data regimes, and study all the combinations on the WLASL2000 dataset. Our findings reveal that MaskFeat achieves performance superior to pose-based and supervised video models, with a top-1 accuracy of 79.02% on gloss-based WLASL2000. Furthermore, we analyze these models' ability to produce representations of ASL signs using linear probing on diverse phonological features. This study underscores the value of architecture and pre-training task choices in ISLR. Specifically, our results on WLASL2000 highlight the power of masked reconstruction pre-training, and our linear probing results demonstrate the importance of hierarchical vision transformers for sign language representation.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 37 of its 48 references that are in the map. First posted on arXiv 2 Sep 2023.

Builds on

37 of 48 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  7. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  8. 2021Video Swin Transformercited by 1,090
  9. 2021Multiscale Vision Transformerscited by 896
  10. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  11. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  12. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
25 more
  1. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  2. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  3. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  4. 2019Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparisoncited by 217
  5. 2017Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognitioncited by 178
  6. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  7. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
  8. 2021VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examplescited by 154
  9. 2020BSL-1K: Scaling up co-articulated sign language recognition using mouthing cuescited by 106
  10. 2021Skeleton Aware Multi-modal Sign Language Recognitioncited by 101
  11. 2022Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learningcited by 90
  12. 2022A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translationcited by 80
  13. 2022Open-Domain Sign Language Translation Learned from Online Videocited by 72
  14. 2021Including Signed Languages in Natural Language Processingcited by 67
  15. 2021Self-supervised Video Transformercited by 64
  16. 2021VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learningcited by 56
  17. 2023Natural Language-Assisted Sign Language Recognitioncited by 46
  18. 2021Video Contrastive Learning with Global Contextcited by 40
  19. 2022Masked Motion Encoding for Self-Supervised Video Representation Learningcited by 39
  20. 2021Read and Attend: Temporal Localisation in Sign Language Videoscited by 34
  21. 2021Sign Language Recognition via Skeleton-Aware Multi-Model Ensemblecited by 16
  22. 2022Predicting within and across language phoneme recognition performance of self-supervised learning speech pre-trained modelscited by 7
  23. 2022WLASL-LEX: a Dataset for Recognising Phonological Properties in American Sign Languagecited by 5
  24. 2023Improving Sign Recognition with Phonologycited by 3
  25. 2023Fine-tuning of sign language recognition models: a technical reportcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 4.

6 papers.

  1. 2024SignMusketeers: An Efficient Multi-Stream Approach for Sign Language Translation at Scalecited by 18
  2. 2024Improvement in Sign Language Translation Using Text CTC Alignmentcited by 5
  3. 2025SignBart -- New approach with the skeleton sequence for Isolated Sign language Recognitioncited by 1
  4. 2025Enhancing Mathematics Learning for Hard-of-Hearing Students Through Real-Time Palestinian Sign Language Recognition: A New Datasetcited by 0
  5. 2025Fine-Tuning Video Transformers for Word-Level Bangla Sign Language: A Comparative Analysis for Classification Taskscited by 0
  6. 2025Isolated Sign Language Recognition with Segmentation and Pose Estimationcited by 0