mlsanityData to 7 Oct 2026

SignCLIP: Connecting Text and Sign Language by Contrastive Learning

arXiv 2407.01264 ↗First posted 1 Jul 2024cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

We present SignCLIP, which re-purposes CLIP (Contrastive Language-Image Pretraining) to project spoken language text and sign language videos, two classes of natural languages of distinct modalities, into the same space. SignCLIP is an efficient method of learning useful visual representations for sign language processing from large-scale, multilingual video-text pairs, without directly optimizing for a specific task or sign language which is often of limited size. We pretrain SignCLIP on Spreadthesign, a prominent sign language dictionary consisting of ~500 thousand video clips in up to 44 sign languages, and evaluate it with various downstream datasets. SignCLIP discerns in-domain signing with notable text-to-video/video-to-text retrieval accuracy. It also performs competitively for out-of-domain downstream tasks such as isolated sign language recognition upon essential few-shot prompting or fine-tuning. We analyze the latent space formed by the spoken language text and sign language poses, which provides additional linguistic insights. Our code and models are openly available.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 50 of its 87 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

50 of 87 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  10. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  11. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  12. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
38 more
  1. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  2. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  3. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  4. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  5. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 572
  6. 2023CoTracker: It is Better to Track Togethercited by 516
  7. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  8. 2019Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparisoncited by 217
  9. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  10. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  11. 2020Deep Learning-Based Human Pose Estimation: A Surveycited by 205
  12. 2022AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Timecited by 199
  13. 2023Tracking Everything Everywhere All at Oncecited by 190
  14. 2021Improving Sign Language Translation with Monolingual Data by Sign Back-Translationcited by 177
  15. 2020How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Languagecited by 172
  16. 2020BSL-1K: Scaling up co-articulated sign language recognition using mouthing cuescited by 106
  17. 2016Learning Visual N-Grams from Web Datacited by 95
  18. 2019Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspectivecited by 87
  19. 2020A Comprehensive Study of Deep Video Action Recognitioncited by 87
  20. 2024LLMs are Good Sign Language Translatorscited by 76
  21. 2023YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpuscited by 74
  22. 2021Including Signed Languages in Natural Language Processingcited by 67
  23. 2023CVT-SLR: Contrastive Visual-Textual Transformation for Sign Language Recognition with Variational Alignmentcited by 44
  24. 2021Read and Attend: Temporal Localisation in Sign Language Videoscited by 34
  25. 2023CiCo: Domain-Aware Sign Language Retrieval via Cross-Lingual Contrastive Learningcited by 32
  26. 2021OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languagescited by 30
  27. 2024Improving Gloss-free Sign Language Translation by Reducing Representation Densitycited by 30
  28. 2019Fingerspelling recognition in the wild with iterative visual attentioncited by 28
  29. 2021Sign Language Production: A Reviewcited by 26
  30. 2021Evaluating the Immediate Applicability of Pose Estimation for Sign Language Recognitioncited by 25
  31. 2018American Sign Language fingerspelling recognition in the wildcited by 20
  32. 2023JWSign: A Highly Multilingual Corpus of Bible Translations for more Diversity in Sign Language Processingcited by 18
  33. 2022Machine Translation between Spoken Languages and Signed Languages Represented in SignWritingcited by 17
  34. 2023The Sem-Lex Benchmark: Modeling ASL Signs and Their Phonemescited by 17
  35. 2018S3D: Single Shot multi-Span Detector via Fully 3D Convolutional Networkscited by 16
  36. 2024A Tale of Two Languages: Large-Vocabulary Continuous Sign Language Recognition from Spoken Language Supervisioncited by 15
  37. 2022Weakly-supervised Fingerspelling Recognition in British Sign Language Videoscited by 10
  38. 2023Autoregressive Sign Language Production: A Gloss-Free Approach with Discrete Representationscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 9.

20 papers.

Sort:

By area:

  1. 2025Lost in Translation, Found in Embeddings: Sign Language Translation and Alignmentcited by 6
  2. 2025CLIP-SLA: Parameter-Efficient CLIP Adaptation for Continuous Sign Language Recognitioncited by 4
  3. 2025SiLVERScore: Semantically-Aware Embeddings for Sign Language Generation Evaluationcited by 4
  4. 2026MaDiS: Taming Masked Diffusion Language Models for Sign Language Generationcited by 3
  5. 2025Stable Signer: Hierarchical Sign Language Generative Modelcited by 2
  6. 2025Segment, Embed, and Align: A Universal Recipe for Aligning Subtitles to Signingcited by 2
  7. 2026Sign Language Question Answering: A New Task, Benchmark, and Baseline for Sign Language Understandingcited by 2
  8. 2025Leveraging Large Language Models for Accurate Sign Language Translation in Low-Resource Scenarioscited by 1
  9. 2025How Pragmatics Shape Articulation: A Computational Case Study in STEM ASL Discoursecited by 1
  10. 2026BackTranslation2.0 -- A Linguistically Motivated Metric to Assess Sign Language Productioncited by 1
  11. 2025Towards Skeletal and Signer Noise Reduction in Sign Language Production via Quaternion-Based Pose Encoding and Contrastive Learningcited by 0
  12. 2025SSL-SLR: Self-Supervised Representation Learning for Sign Language Recognitioncited by 0
8 more
  1. 2025Understanding InfoNCE: Transition Probability Matrix Induced Feature Clusteringcited by 0
  2. 2025MHB: Multimodal Handshape-aware Boundary Detection for Continuous Sign Language Recognitioncited by 0
  3. 2026SyriSign: A Parallel Corpus for Arabic Text to Syrian Arabic Sign Language Translationcited by 0
  4. 2026DualAnchor: Preserving Language Priors and Improving Lexical Fidelity in Gloss-Free Sign Language Translationcited by 0
  5. 2026SignSeek: Learning Transferable Representations for Sign Dictionary Retrievalcited by 0
  6. 2026A Reverse Sign Language Dictionary: Open-Vocabulary Sign Recognition from Continuous Signing via Video Captioning and Description Retrievalcited by 0
  7. 2026SignDino: Self-Supervised Sign Language Representation Learning via Temporal-Axis Self-Distillationcited by 0
  8. 2026Machine Translation for Sign Languagescited by 0