# Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces

arXiv: https://arxiv.org/abs/1805.07467 · HTML: https://mlsanity.com/paper/1805.07467/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 May 2018.

## Built on by (most cited first, 30 of 34)

- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition](https://mlsanity.com/paper/2109.06870/) (2021)
- [Audio Self-supervised Learning: A Survey](https://mlsanity.com/paper/2203.01205/) (2022)
- [Unified Speech-Text Pre-training for Speech Translation and Recognition](https://mlsanity.com/paper/2204.05409/) (2022)
- [PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition](https://mlsanity.com/paper/2106.05933/) (2021)
- [Towards End-to-end Unsupervised Speech Recognition](https://mlsanity.com/paper/2204.02492/) (2022)
- [Vision+X: A Survey on Multimodal Learning in the Light of Data](https://mlsanity.com/paper/2210.02884/) (2022)
- [BSTC: A Large-Scale Chinese-English Speech Translation Dataset](https://mlsanity.com/paper/2104.03575/) (2021)
- [Non-Parametric Domain Adaptation for End-to-End Speech Translation](https://mlsanity.com/paper/2205.11211/) (2022)
- [Speech Sequence Embeddings using Nearest Neighbors Contrastive Learning](https://mlsanity.com/paper/2204.05148/) (2022)
- [Distilling a Pretrained Language Model to a Multilingual ASR Model](https://mlsanity.com/paper/2206.12638/) (2022)
- [On the Difficulty of Segmenting Words with Attention](https://mlsanity.com/paper/2109.10107/) (2021)
- [Unsupervised Automatic Speech Recognition: A Review](https://mlsanity.com/paper/2106.04897/) (2021)
- [An Analysis of Semantically-Aligned Speech-Text Embeddings](https://mlsanity.com/paper/2204.01235/) (2022)
- [Improving Cascaded Unsupervised Speech Translation with Denoising Back-translation](https://mlsanity.com/paper/2305.07455/) (2023)
- [Speechless: Speech Instruction Training Without Speech for Low Resource Languages](https://mlsanity.com/paper/2505.17417/) (2025)
- [Align before Search: Aligning Ads Image to Text for Accurate Cross-Modal Sponsored Search](https://mlsanity.com/paper/2309.16141/) (2023)
- [CLASP: Contrastive Language-Speech Pretraining for Multilingual Multimodal Information Retrieval](https://mlsanity.com/paper/2412.13071/) (2024)
- [Chunk Based Speech Pre-training with High Resolution Finite Scalar Quantization](https://mlsanity.com/paper/2509.15579/) (2025)
- [Contextual Text Embeddings for Twi](https://mlsanity.com/paper/2103.15963/) (2021)
- [Information Theory-Guided Heuristic Progressive Multi-View Coding](https://mlsanity.com/paper/2109.02344/) (2021)
- [QuoteKG: A Multilingual Knowledge Graph of Quotes](https://mlsanity.com/paper/2207.09562/) (2022)
- [Learning Phone Recognition from Unpaired Audio and Phone Sequences Based on Generative Adversarial Network](https://mlsanity.com/paper/2207.14568/) (2022)
- [Variational Connectionist Temporal Classification for Order-Preserving Sequence Modeling](https://mlsanity.com/paper/2309.11983/) (2023)
- [Unsupervised Mismatch Localization in Cross-Modal Sequential Data with Application to Mispronunciations Localization](https://mlsanity.com/paper/2205.02670/) (2022)
- [Homophone Reveals the Truth: A Reality Check for Speech2Vec](https://mlsanity.com/paper/2209.10791/) (2022)
- [Information Theory-Guided Heuristic Progressive Multi-View Coding](https://mlsanity.com/paper/2308.10522/) (2023)
- [Improving End-to-End Speech Processing by Efficient Text Data Utilization with Latent Synthesis](https://mlsanity.com/paper/2310.05374/) (2023)

## Abstract (arXiv)

Recent research has shown that word embedding spaces learned from text corpora of different languages can be aligned without any parallel data supervision. Inspired by the success in unsupervised cross-lingual word embeddings, in this paper we target learning a cross-modal alignment between the embedding spaces of speech and text learned from corpora of their respective modalities in an unsupervised fashion. The proposed framework learns the individual speech and text embedding spaces, and attempts to align the two spaces via adversarial training, followed by a refinement procedure. We show how our framework could be used to perform spoken word classification and translation, and the results on these two tasks demonstrate that the performance of our unsupervised alignment approach is comparable to its supervised counterpart. Our framework is especially useful for developing automatic speech recognition (ASR) and speech-to-text translation systems for low- or zero-resource languages, which have little parallel audio-text data for training modern supervised ASR and speech-to-text translation models, but account for the majority of the languages spoken across the world.