# SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model

arXiv: https://arxiv.org/abs/2210.00705 · HTML: https://mlsanity.com/paper/2210.00705/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 50 references that are in the map.
- First posted on arXiv 3 Oct 2022.

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders](https://mlsanity.com/paper/1910.12638/) (2019)
- [TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech](https://mlsanity.com/paper/2007.06028/) (2020)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [Self-supervised Learning with Random-projection Quantizer for Speech Recognition](https://mlsanity.com/paper/2202.01855/) (2022)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT](https://mlsanity.com/paper/2110.01900/) (2021)
- [Multi-task self-supervised learning for Robust Speech Recognition](https://mlsanity.com/paper/2001.09239/) (2020)
- [Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input](https://mlsanity.com/paper/1804.01452/) (2018)
- [SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities](https://mlsanity.com/paper/2203.06849/) (2022)
- [Vector-Quantized Autoregressive Predictive Coding](https://mlsanity.com/paper/2005.08392/) (2020)
- [Deep Multimodal Semantic Embeddings for Speech and Images](https://mlsanity.com/paper/1511.03690/) (2015)
- [Text-Free Image-to-Speech Synthesis Using Learned Segmental Units](https://mlsanity.com/paper/2012.15454/) (2020)
- [LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT](https://mlsanity.com/paper/2203.15610/) (2022)
- [Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies](https://mlsanity.com/paper/2011.00406/) (2020)

## Built on by (most cited first, 29 of 29)

- [Audio Mamba: Bidirectional State Space Model for Audio Representation Learning](https://mlsanity.com/paper/2406.03344/) (2024)
- [SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2210.08634/) (2022)
- [Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Language](https://mlsanity.com/paper/2406.05629/) (2024)
- [Can Graph Neural Networks Learn Language with Extremely Weak Text Supervision?](https://mlsanity.com/paper/2412.08174/) (2024)
- [Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Model](https://mlsanity.com/paper/2305.11435/) (2023)
- [M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrieval](https://mlsanity.com/paper/2211.01180/) (2022)
- [TimbreCLIP: Connecting Timbre to Text and Images](https://mlsanity.com/paper/2211.11225/) (2022)
- [Interface Design for Self-Supervised Speech Models](https://mlsanity.com/paper/2406.12209/) (2024)
- [Training Transitive and Commutative Multimodal Transformers with LoReTTa](https://mlsanity.com/paper/2305.14243/) (2023)
- [Towards Practical and Efficient Image-to-Speech Captioning with Vision-Language Pre-training and Multi-modal Tokens](https://mlsanity.com/paper/2309.08531/) (2023)
- [You Only Speak Once to See](https://mlsanity.com/paper/2409.18372/) (2024)
- [Vision-Speech Models: Teaching Speech Models to Converse about Images](https://mlsanity.com/paper/2503.15633/) (2025)
- [Visually Grounded Speech Models have a Mutual Exclusivity Bias](https://mlsanity.com/paper/2403.13922/) (2024)
- [Measuring Sound Symbolism in Audio-visual Models](https://mlsanity.com/paper/2409.12306/) (2024)
- [SpeechCLIP+: Self-supervised multi-task representation learning for speech via CLIP and speech-image data](https://mlsanity.com/paper/2402.06959/) (2024)
- [Translating speech with just images](https://mlsanity.com/paper/2406.07133/) (2024)
- [CLASP: Contrastive Language-Speech Pretraining for Multilingual Multimodal Information Retrieval](https://mlsanity.com/paper/2412.13071/) (2024)
- [End-to-end Contrastive Language-Speech Pretraining Model For Long-form Spoken Question Answering](https://mlsanity.com/paper/2511.09282/) (2025)
- [Integrating Self-supervised Speech Model with Pseudo Word-level Targets from Visually-grounded Speech Model](https://mlsanity.com/paper/2402.05819/) (2024)
- [Coarse-to-fine Alignment Makes Better Speech-image Retrieval](https://mlsanity.com/paper/2408.13119/) (2024)
- [Expanding Event Modality Applications through a Robust CLIP-Based Encoder](https://mlsanity.com/paper/2412.03093/) (2024)
- [Leveraging Audio-Visual Data to Reduce the Multilingual Gap in Self-Supervised Speech Models](https://mlsanity.com/paper/2509.17523/) (2025)
- [ParaSpeechCLAP: A Dual-Encoder Speech-Text Model for Rich Stylistic Language-Audio Pretraining](https://mlsanity.com/paper/2603.28737/) (2026)
- [SCRAPS: Speech Contrastive Representations of Acoustic and Phonetic Spaces](https://mlsanity.com/paper/2307.12445/) (2023)
- [Cross-Modal Coordination Across a Diverse Set of Input Modalities](https://mlsanity.com/paper/2401.16347/) (2024)
- [BrewCLIP: A Bifurcated Representation Learning Framework for Audio-Visual Retrieval](https://mlsanity.com/paper/2408.10383/) (2024)
- [Cross-Modal Denoising: A Novel Training Paradigm for Enhancing Speech-Image Retrieval](https://mlsanity.com/paper/2408.13705/) (2024)
- [Unifying Model and Layer Fusion for Speech Foundation Models](https://mlsanity.com/paper/2511.08389/) (2025)
- [Speak the Art: A Direct Speech to Image Generation Framework](https://mlsanity.com/paper/2601.00827/) (2025)

## Abstract (arXiv)

Data-driven speech processing models usually perform well with a large amount of text supervision, but collecting transcribed speech data is costly. Therefore, we propose SpeechCLIP, a novel framework bridging speech and text through images to enhance speech models without transcriptions. We leverage state-of-the-art pre-trained HuBERT and CLIP, aligning them via paired images and spoken captions with minimal fine-tuning. SpeechCLIP outperforms prior state-of-the-art on image-speech retrieval and performs zero-shot speech-text retrieval without direct supervision from transcriptions. Moreover, SpeechCLIP can directly retrieve semantically related keywords from speech.