# Audio-Linguistic Embeddings for Spoken Sentences

arXiv: https://arxiv.org/abs/1902.07817 · HTML: https://mlsanity.com/paper/1902.07817/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Feb 2019.

## Built on by (most cited first, 15 of 15)

- [Audio Self-supervised Learning: A Survey](https://mlsanity.com/paper/2203.01205/) (2022)
- [Audio Transformers](https://mlsanity.com/paper/2105.00335/) (2021)
- [Acoustic word embeddings for zero-resource languages using self-supervised contrastive learning and multilingual adaptation](https://mlsanity.com/paper/2103.10731/) (2021)
- [A Generative Model for Raw Audio Using Transformer Architectures](https://mlsanity.com/paper/2106.16036/) (2021)
- [CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations](https://mlsanity.com/paper/2109.00181/) (2021)
- [On Compressing Sequences for Self-Supervised Speech Models](https://mlsanity.com/paper/2210.07189/) (2022)
- [Multilingual transfer of acoustic word embeddings improves when training on languages related to the target zero-resource language](https://mlsanity.com/paper/2106.12834/) (2021)
- [Deception detection in text and its relation to the cultural dimension of individualism/collectivism](https://mlsanity.com/paper/2105.12530/) (2021)
- [Attention is All You Need? Good Embeddings with Statistics are enough:Large Scale Audio Understanding without Transformers/ Convolutions/ BERTs/ Mixers/ Attention/ RNNs or ....](https://mlsanity.com/paper/2110.03183/) (2021)
- [Conformer LLMs -- Convolution Augmented Large Language Models](https://mlsanity.com/paper/2307.00461/) (2023)
- [One-Shot Acoustic Matching Of Audio Signals -- Learning to Hear Music In Any Room/ Concert Hall](https://mlsanity.com/paper/2210.15750/) (2022)
- [Content Adaptive Front End For Audio Classification](https://mlsanity.com/paper/2303.10446/) (2023)
- [Topic Identification For Spontaneous Speech: Enriching Audio Features With Embedded Linguistic Information](https://mlsanity.com/paper/2307.11450/) (2023)
- [Fusion approaches for emotion recognition from speech using acoustic and text-based features](https://mlsanity.com/paper/2403.18635/) (2024)
- [Forewarned is Forearmed: When Non-Sequential Embedding Turns Into an Anomaly Detector](https://mlsanity.com/paper/2606.30196/) (2026)

## Abstract (arXiv)

We propose spoken sentence embeddings which capture both acoustic and linguistic content. While existing works operate at the character, phoneme, or word level, our method learns long-term dependencies by modeling speech at the sentence level. Formulated as an audio-linguistic multitask learning problem, our encoder-decoder model simultaneously reconstructs acoustic and natural language features from audio. Our results show that spoken sentence embeddings outperform phoneme and word-level baselines on speech recognition and emotion recognition tasks. Ablation studies show that our embeddings can better model high-level acoustic concepts while retaining linguistic content. Overall, our work illustrates the viability of generic, multi-modal sentence embeddings for spoken language understanding.