# Acoustically Grounded Word Embeddings for Improved Acoustics-to-Word Speech Recognition

arXiv: https://arxiv.org/abs/1903.12306 · HTML: https://mlsanity.com/paper/1903.12306/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Mar 2019.

## Built on by (most cited first, 15 of 15)

- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [What Do Self-Supervised Speech Models Know About Words?](https://mlsanity.com/paper/2307.00162/) (2023)
- [Acoustic word embeddings for zero-resource languages using self-supervised contrastive learning and multilingual adaptation](https://mlsanity.com/paper/2103.10731/) (2021)
- [Do Acoustic Word Embeddings Capture Phonological Similarity? An Empirical Study](https://mlsanity.com/paper/2106.08686/) (2021)
- [Multilingual transfer of acoustic word embeddings improves when training on languages related to the target zero-resource language](https://mlsanity.com/paper/2106.12834/) (2021)
- [AdaMS: Deep Metric Learning with Adaptive Margin and Adaptive Scale for Acoustic Word Discrimination](https://mlsanity.com/paper/2210.14564/) (2022)
- [Analyzing the Representational Geometry of Acoustic Word Embeddings](https://mlsanity.com/paper/2301.03012/) (2023)
- [Investigating the 'Autoencoder Behavior' in Speech Self-Supervised Models: a focus on HuBERT's Pretraining](https://mlsanity.com/paper/2405.08402/) (2024)
- [Word Order Does Not Matter For Speech Recognition](https://mlsanity.com/paper/2110.05994/) (2021)
- [Integrating Form and Meaning: A Multi-Task Learning Model for Acoustic Word Embeddings](https://mlsanity.com/paper/2209.06633/) (2022)
- [Using Pause Information for More Accurate Entity Recognition](https://mlsanity.com/paper/2109.13222/) (2021)
- [Timestamped Embedding-Matching Acoustic-to-Word CTC ASR](https://mlsanity.com/paper/2306.11473/) (2023)
- [An Empirical Analysis of Speech Self-Supervised Learning at Multiple Resolutions](https://mlsanity.com/paper/2410.23955/) (2024)

## Abstract (arXiv)

Direct acoustics-to-word (A2W) systems for end-to-end automatic speech recognition are simpler to train, and more efficient to decode with, than sub-word systems. However, A2W systems can have difficulties at training time when data is limited, and at decoding time when recognizing words outside the training vocabulary. To address these shortcomings, we investigate the use of recently proposed acoustic and acoustically grounded word embedding techniques in A2W systems. The idea is based on treating the final pre-softmax weight matrix of an AWE recognizer as a matrix of word embedding vectors, and using an externally trained set of word embeddings to improve the quality of this matrix. In particular we introduce two ideas: (1) Enforcing similarity at training time between the external embeddings and the recognizer weights, and (2) using the word embeddings at test time for predicting out-of-vocabulary words. Our word embedding model is acoustically grounded, that is it is learned jointly with acoustic embeddings so as to encode the words' acoustic-phonetic content; and it is parametric, so that it can embed any arbitrary (potentially out-of-vocabulary) sequence of characters. We find that both techniques improve the performance of an A2W recognizer on conversational telephone speech.