# Do We Still Need Automatic Speech Recognition for Spoken Language Understanding?

arXiv: https://arxiv.org/abs/2111.14842 · HTML: https://mlsanity.com/paper/2111.14842/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 18 references that are in the map.
- First posted on arXiv 29 Nov 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders](https://mlsanity.com/paper/1910.12638/) (2019)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training](https://mlsanity.com/paper/2104.01027/) (2021)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks](https://mlsanity.com/paper/1904.03416/) (2019)
- [Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining](https://mlsanity.com/paper/2010.13826/) (2020)
- [Speech-language Pre-training for End-to-end Spoken Language Understanding](https://mlsanity.com/paper/2102.06283/) (2021)
- [On Scaling Contrastive Representations for Low-Resource Speech Recognition](https://mlsanity.com/paper/2102.00850/) (2021)

## Built on by (most cited first, 5 of 5)

- [On the Use of External Data for Spoken Named Entity Recognition](https://mlsanity.com/paper/2112.07648/) (2021)
- [Bridging Speech and Textual Pre-trained Models with Unsupervised ASR](https://mlsanity.com/paper/2211.03025/) (2022)
- [Finstreder: Simple and fast Spoken Language Understanding with Finite State Transducers using modern Speech-to-Text models](https://mlsanity.com/paper/2206.14589/) (2022)
- [Multi-Variant Consistency based Self-supervised Learning for Robust Automatic Speech Recognition](https://mlsanity.com/paper/2112.12522/) (2021)
- [Sentence Embedder Guided Utterance Encoder (SEGUE) for Spoken Language Understanding](https://mlsanity.com/paper/2305.12301/) (2023)

## Abstract (arXiv)

Spoken language understanding (SLU) tasks are usually solved by first transcribing an utterance with automatic speech recognition (ASR) and then feeding the output to a text-based model. Recent advances in self-supervised representation learning for speech data have focused on improving the ASR component. We investigate whether representation learning for speech has matured enough to replace ASR in SLU. We compare learned speech features from wav2vec 2.0, state-of-the-art ASR transcripts, and the ground truth text as input for a novel speech-based named entity recognition task, a cardiac arrest detection task on real-world emergency calls and two existing SLU benchmarks. We show that learned speech features are superior to ASR transcripts on three classification tasks. For machine translation, ASR transcripts are still the better choice. We highlight the intrinsic robustness of wav2vec 2.0 representations to out-of-vocabulary words as key to better performance.