# Audio Word2Vec: Unsupervised Learning of Audio Segment Representations using Sequence-to-sequence Autoencoder

arXiv: https://arxiv.org/abs/1603.00982 · HTML: https://mlsanity.com/paper/1603.00982/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Mar 2016.

## Built on by (most cited first, 30 of 34)

- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training](https://mlsanity.com/paper/2110.10329/) (2021)
- [A Survey on Time-Series Pre-Trained Models](https://mlsanity.com/paper/2305.10716/) (2023)
- [Deep Spoken Keyword Spotting: An Overview](https://mlsanity.com/paper/2111.10592/) (2021)
- [Broad-UNet: Multi-scale feature learning for nowcasting tasks](https://mlsanity.com/paper/2102.06442/) (2021)
- [TVLT: Textless Vision-Language Transformer](https://mlsanity.com/paper/2209.14156/) (2022)
- [Utilizing Self-supervised Representations for MOS Prediction](https://mlsanity.com/paper/2104.03017/) (2021)
- [Matching Latent Encoding for Audio-Text based Keyword Spotting](https://mlsanity.com/paper/2306.05245/) (2023)
- [A Brief Overview of Unsupervised Neural Speech Representation Learning](https://mlsanity.com/paper/2203.01829/) (2022)
- [Speech Sequence Embeddings using Nearest Neighbors Contrastive Learning](https://mlsanity.com/paper/2204.05148/) (2022)
- [Interpreting intermediate convolutional layers of generative CNNs trained on waveforms](https://mlsanity.com/paper/2104.09489/) (2021)
- [Unsupervised Automatic Speech Recognition: A Review](https://mlsanity.com/paper/2106.04897/) (2021)
- [Modeling speech recognition and synthesis simultaneously: Encoding and decoding lexical and sublexical semantic information into speech with no direct access to speech data](https://mlsanity.com/paper/2203.11476/) (2022)
- [Towards Proper Contrastive Self-supervised Learning Strategies For Music Audio Representation](https://mlsanity.com/paper/2207.04471/) (2022)
- [Towards hate speech detection in low-resource languages: Comparing ASR to acoustic word embeddings on Wolof and Swahili](https://mlsanity.com/paper/2306.00410/) (2023)
- [Exploring How Generative Adversarial Networks Learn Phonological Representations](https://mlsanity.com/paper/2305.12501/) (2023)
- [Cetacean Translation Initiative: a roadmap to deciphering the communication of sperm whales](https://mlsanity.com/paper/2104.08614/) (2021)
- [Interpreting intermediate convolutional layers in unsupervised acoustic word classification](https://mlsanity.com/paper/2110.02375/) (2021)
- [Visually Grounded Speech Models have a Mutual Exclusivity Bias](https://mlsanity.com/paper/2403.13922/) (2024)
- [Auto-KWS 2021 Challenge: Task, Datasets, and Baselines](https://mlsanity.com/paper/2104.00513/) (2021)
- [A Quantitative Approach to Understand Self-Supervised Models as Cross-lingual Feature Extractors](https://mlsanity.com/paper/2311.15954/) (2023)
- [Modeling Dynamics of Facial Behavior for Mental Health Assessment](https://mlsanity.com/paper/2108.09934/) (2021)
- [Learning Phone Recognition from Unpaired Audio and Phone Sequences Based on Generative Adversarial Network](https://mlsanity.com/paper/2207.14568/) (2022)
- [Improving Speech Representation Learning via Speech-level and Phoneme-level Masking Approach](https://mlsanity.com/paper/2210.13805/) (2022)
- [Supervised Acoustic Embeddings And Their Transferability Across Languages](https://mlsanity.com/paper/2301.01020/) (2023)
- [Acoustic models of Brazilian Portuguese Speech based on Neural Transformers](https://mlsanity.com/paper/2312.09265/) (2023)
- [Unsupervised Discriminative Learning of Sounds for Audio Event Classification](https://mlsanity.com/paper/2105.09279/) (2021)
- [A Novel Semi-supervised Framework for Call Center Agent Malpractice Detection via Neural Feature Learning](https://mlsanity.com/paper/2106.02433/) (2021)
- [On Training Targets and Activation Functions for Deep Representation Learning in Text-Dependent Speaker Verification](https://mlsanity.com/paper/2201.06426/) (2022)

## Abstract (arXiv)

The vector representations of fixed dimensionality for words (in text) offered by Word2Vec have been shown to be very useful in many application scenarios, in particular due to the semantic information they carry. This paper proposes a parallel version, the Audio Word2Vec. It offers the vector representations of fixed dimensionality for variable-length audio segments. These vector representations are shown to describe the sequential phonetic structures of the audio segments to a good degree, with very attractive real world applications such as query-by-example Spoken Term Detection (STD). In this STD application, the proposed approach significantly outperformed the conventional Dynamic Time Warping (DTW) based approaches at significantly lower computation requirements. We propose unsupervised learning of Audio Word2Vec from audio data without human annotation using Sequence-to-sequence Audoencoder (SA). SA consists of two RNNs equipped with Long Short-Term Memory (LSTM) units: the first RNN (encoder) maps the input audio sequence into a vector representation of fixed dimensionality, and the second RNN (decoder) maps the representation back to the input audio sequence. The two RNNs are jointly trained by minimizing the reconstruction error. Denoising Sequence-to-sequence Autoencoder (DSA) is furthered proposed offering more robust learning.