# Segmental Audio Word2Vec: Representing Utterances as Sequences of Vectors with Applications in Spoken Term Detection

arXiv: https://arxiv.org/abs/1808.02228 · HTML: https://mlsanity.com/paper/1808.02228/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2018.

## Built on by (most cited first, 17 of 17)

- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Towards End-to-end Unsupervised Speech Recognition](https://mlsanity.com/paper/2204.02492/) (2022)
- [Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoring](https://mlsanity.com/paper/2202.11929/) (2022)
- [Utilizing Self-supervised Representations for MOS Prediction](https://mlsanity.com/paper/2104.03017/) (2021)
- [Acoustic word embeddings for zero-resource languages using self-supervised contrastive learning and multilingual adaptation](https://mlsanity.com/paper/2103.10731/) (2021)
- [Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Model](https://mlsanity.com/paper/2305.11435/) (2023)
- [Multilingual transfer of acoustic word embeddings improves when training on languages related to the target zero-resource language](https://mlsanity.com/paper/2106.12834/) (2021)
- [Unsupervised Word Discovery: Boundary Detection with Clustering vs. Dynamic Programming](https://mlsanity.com/paper/2409.14486/) (2024)
- [Unsupervised Automatic Speech Recognition: A Review](https://mlsanity.com/paper/2106.04897/) (2021)
- [Spoken Term Detection and Relevance Score Estimation using Dot-Product of Pronunciation Embeddings](https://mlsanity.com/paper/2210.11895/) (2022)
- [Visually grounded few-shot word learning in low-resource settings](https://mlsanity.com/paper/2306.11371/) (2023)
- [REBORN: Reinforcement-Learned Boundary Segmentation with Iterative Training for Unsupervised ASR](https://mlsanity.com/paper/2402.03988/) (2024)
- [Deep LSTM Spoken Term Detection using Wav2Vec 2.0 Recognizer](https://mlsanity.com/paper/2210.11885/) (2022)
- [Visually Grounded Speech Models have a Mutual Exclusivity Bias](https://mlsanity.com/paper/2403.13922/) (2024)
- [Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddings](https://mlsanity.com/paper/2210.12857/) (2022)
- [Visually grounded few-shot word acquisition with fewer shots](https://mlsanity.com/paper/2305.15937/) (2023)
- [Adding Connectionist Temporal Summarization into Conformer to Improve Its Decoder Efficiency For Speech Recognition](https://mlsanity.com/paper/2204.03889/) (2022)

## Abstract (arXiv)

While Word2Vec represents words (in text) as vectors carrying semantic information, audio Word2Vec was shown to be able to represent signal segments of spoken words as vectors carrying phonetic structure information. Audio Word2Vec can be trained in an unsupervised way from an unlabeled corpus, except the word boundaries are needed. In this paper, we extend audio Word2Vec from word-level to utterance-level by proposing a new segmental audio Word2Vec, in which unsupervised spoken word boundary segmentation and audio Word2Vec are jointly learned and mutually enhanced, so an utterance can be directly represented as a sequence of vectors carrying phonetic structure information. This is achieved by a segmental sequence-to-sequence autoencoder (SSAE), in which a segmentation gate trained with reinforcement learning is inserted in the encoder. Experiments on English, Czech, French and German show very good performance in both unsupervised spoken word segmentation and spoken term detection applications (significantly better than frame-based DTW).