mlsanityData to 7 Oct 2026

SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model

arXiv 2210.00705 ↗First posted 3 Oct 2022cs.CL cs.SD eess.AScited by 29

Not on the map. Why?

Abstract (arXiv)

Data-driven speech processing models usually perform well with a large amount of text supervision, but collecting transcribed speech data is costly. Therefore, we propose SpeechCLIP, a novel framework bridging speech and text through images to enhance speech models without transcriptions. We leverage state-of-the-art pre-trained HuBERT and CLIP, aligning them via paired images and spoken captions with minimal fine-tuning. SpeechCLIP outperforms prior state-of-the-art on image-speech retrieval and performs zero-shot speech-text retrieval without direct supervision from transcriptions. Moreover, SpeechCLIP can directly retrieve semantically related keywords from speech.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 47 of its 50 references that are in the map. First posted on arXiv 3 Oct 2022.

Builds on

47 of 50 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  8. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  9. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  10. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  11. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  12. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
35 more
  1. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  2. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  3. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  4. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 210
  5. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  6. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  7. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  8. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  9. 2021Unsupervised Speech Recognitioncited by 147
  10. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  11. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  12. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  13. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 76
  14. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  15. 2015Deep Multimodal Semantic Embeddings for Speech and Imagescited by 56
  16. 2020Text-Free Image-to-Speech Synthesis Using Learned Segmental Unitscited by 50
  17. 2020Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependenciescited by 47
  18. 2021LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speechcited by 46
  19. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 45
  20. 2022Towards End-to-end Unsupervised Speech Recognitioncited by 44
  21. 2022Word Discovery in Visually Grounded, Self-Supervised Speech Modelscited by 37
  22. 2019Large-scale representation learning from visually grounded untranscribed speechcited by 33
  23. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
  24. 2022SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representationcited by 32
  25. 2018Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speechcited by 26
  26. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  27. 2021Fast-Slow Transformer for Visually Grounding Speechcited by 23
  28. 2017Semantic speech retrieval with a visually grounded model of untranscribed speechcited by 20
  29. 2018Visually grounded cross-lingual keyword spotting in speechcited by 19
  30. 2021Talk, Don't Write: A Study of Direct Speech-Based Image Retrievalcited by 19
  31. 2019Transfer Learning from Audio-Visual Grounding to Speech Recognitioncited by 16
  32. 2020Catplayinginthesnow: Impact of Prior Segmentation on a Model of Visually Grounded Speechcited by 9
  33. 2019On the Contributions of Visual and Textual Supervision in Low-Resource Semantic Speech Retrievalcited by 8
  34. 2021Evaluation of Audio-Visual Alignments in Visually Grounded Speech Modelscited by 8
  35. 2022Keyword localisation in untranscribed speech using visually grounded speech modelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 4; 2024: 16; 2025: 5; 2026: 1.

29 papers.

Sort:

By area:

  1. 2024Audio Mamba: Bidirectional State Space Model for Audio Representation Learningcited by 36
  2. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  3. 2024Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Languagecited by 17
  4. 2024Can Graph Neural Networks Learn Language with Extremely Weak Text Supervision?cited by 13
  5. 2023Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Modelcited by 11
  6. 2022M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrievalcited by 9
  7. 2022TimbreCLIP: Connecting Timbre to Text and Imagescited by 5
  8. 2024Interface Design for Self-Supervised Speech Modelscited by 5
  9. 2023Training Transitive and Commutative Multimodal Transformers with LoReTTacited by 4
  10. 2023Towards Practical and Efficient Image-to-Speech Captioning with Vision-Language Pre-training and Multi-modal Tokenscited by 4
  11. 2024You Only Speak Once to Seecited by 4
  12. 2025Vision-Speech Models: Teaching Speech Models to Converse about Imagescited by 4
17 more
  1. 2024Visually Grounded Speech Models have a Mutual Exclusivity Biascited by 3
  2. 2024Measuring Sound Symbolism in Audio-visual Modelscited by 3
  3. 2024SpeechCLIP+: Self-supervised multi-task representation learning for speech via CLIP and speech-image datacited by 2
  4. 2024Translating speech with just imagescited by 2
  5. 2024CLASP: Contrastive Language-Speech Pretraining for Multilingual Multimodal Information Retrievalcited by 2
  6. 2025End-to-end Contrastive Language-Speech Pretraining Model For Long-form Spoken Question Answeringcited by 2
  7. 2024Integrating Self-supervised Speech Model with Pseudo Word-level Targets from Visually-grounded Speech Modelcited by 1
  8. 2024Coarse-to-fine Alignment Makes Better Speech-image Retrievalcited by 1
  9. 2024Expanding Event Modality Applications through a Robust CLIP-Based Encodercited by 1
  10. 2025Leveraging Audio-Visual Data to Reduce the Multilingual Gap in Self-Supervised Speech Modelscited by 1
  11. 2026ParaSpeechCLAP: A Dual-Encoder Speech-Text Model for Rich Stylistic Language-Audio Pretrainingcited by 1
  12. 2023SCRAPS: Speech Contrastive Representations of Acoustic and Phonetic Spacescited by 0
  13. 2024Cross-Modal Coordination Across a Diverse Set of Input Modalitiescited by 0
  14. 2024BrewCLIP: A Bifurcated Representation Learning Framework for Audio-Visual Retrievalcited by 0
  15. 2024Cross-Modal Denoising: A Novel Training Paradigm for Enhancing Speech-Image Retrievalcited by 0
  16. 2025Unifying Model and Layer Fusion for Speech Foundation Modelscited by 0
  17. 2025Speak the Art: A Direct Speech to Image Generation Frameworkcited by 0