# Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech

arXiv: https://arxiv.org/abs/1803.08976 · HTML: https://mlsanity.com/paper/1803.08976/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Mar 2018.

## Built on by (most cited first, 30 of 52)

- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing](https://mlsanity.com/paper/2110.07205/) (2021)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2109.13226/) (2021)
- [Simple and Effective Zero-shot Cross-lingual Phoneme Recognition](https://mlsanity.com/paper/2109.11680/) (2021)
- [Audio Self-supervised Learning: A Survey](https://mlsanity.com/paper/2203.01205/) (2022)
- [A Survey on Time-Series Pre-Trained Models](https://mlsanity.com/paper/2305.10716/) (2023)
- [LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT](https://mlsanity.com/paper/2203.15610/) (2022)
- [The Zero Resource Speech Challenge 2021: Spoken language modelling](https://mlsanity.com/paper/2104.14700/) (2021)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challenge](https://mlsanity.com/paper/2210.15759/) (2022)
- [Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT](https://mlsanity.com/paper/2102.07594/) (2021)
- [CLSRIL-23: Cross Lingual Speech Representations for Indic Languages](https://mlsanity.com/paper/2107.07402/) (2021)
- [TVLT: Textless Vision-Language Transformer](https://mlsanity.com/paper/2209.14156/) (2022)
- [Improved Language Identification Through Cross-Lingual Self-Supervised Learning](https://mlsanity.com/paper/2107.04082/) (2021)
- [Acoustic word embeddings for zero-resource languages using self-supervised contrastive learning and multilingual adaptation](https://mlsanity.com/paper/2103.10731/) (2021)
- [Generative Spoken Language Model based on continuous word-sized audio tokens](https://mlsanity.com/paper/2310.05224/) (2023)
- [Capitalization and Punctuation Restoration: a Survey](https://mlsanity.com/paper/2111.10746/) (2021)
- [Maestro-U: Leveraging joint speech-text representation learning for zero supervised speech ASR](https://mlsanity.com/paper/2210.10027/) (2022)
- [ASiT: Local-Global Audio Spectrogram vIsion Transformer for Event Classification](https://mlsanity.com/paper/2211.13189/) (2022)
- [Masked Modeling for Self-supervised Representation Learning on Vision and Beyond](https://mlsanity.com/paper/2401.00897/) (2023)
- [DP-Parse: Finding Word Boundaries from Raw Speech with an Instance Lexicon](https://mlsanity.com/paper/2206.11332/) (2022)
- [A Brief Overview of Unsupervised Neural Speech Representation Learning](https://mlsanity.com/paper/2203.01829/) (2022)
- [Are word boundaries useful for unsupervised language learning?](https://mlsanity.com/paper/2210.02956/) (2022)
- [Multilingual transfer of acoustic word embeddings improves when training on languages related to the target zero-resource language](https://mlsanity.com/paper/2106.12834/) (2021)
- [On the Difficulty of Segmenting Words with Attention](https://mlsanity.com/paper/2109.10107/) (2021)
- [Unsupervised Automatic Speech Recognition: A Review](https://mlsanity.com/paper/2106.04897/) (2021)
- [Relax, it doesn't matter how you get there: A new self-supervised approach for multi-timescale behavior analysis](https://mlsanity.com/paper/2303.08811/) (2023)

## Abstract (arXiv)

In this paper, we propose a novel deep neural network architecture, Speech2Vec, for learning fixed-length vector representations of audio segments excised from a speech corpus, where the vectors contain semantic information pertaining to the underlying spoken words, and are close to other vectors in the embedding space if their corresponding underlying spoken words are semantically similar. The proposed model can be viewed as a speech version of Word2Vec. Its design is based on a RNN Encoder-Decoder framework, and borrows the methodology of skipgrams or continuous bag-of-words for training. Learning word embeddings directly from speech enables Speech2Vec to make use of the semantic information carried by speech that does not exist in plain text. The learned word embeddings are evaluated and analyzed on 13 widely used word similarity benchmarks, and outperform word embeddings learned by Word2Vec from the transcriptions.