# SpeechBERT: An Audio-and-text Jointly Learned Language Model for End-to-end Spoken Question Answering

arXiv: https://arxiv.org/abs/1910.11559 · HTML: https://mlsanity.com/paper/1910.11559/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Oct 2019.

## Built on by (most cited first, 11 of 11)

- [LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks](https://mlsanity.com/paper/2206.06565/) (2022)
- [Self-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answering](https://mlsanity.com/paper/2109.03381/) (2021)
- [End-to-end Spoken Conversational Question Answering: Task, Dataset and Model](https://mlsanity.com/paper/2204.14272/) (2022)
- [ERNIE-GeoL: A Geography-and-Language Pre-trained Model and its Applications in Baidu Maps](https://mlsanity.com/paper/2203.09127/) (2022)
- [HeySQuAD: A Spoken Question Answering Dataset](https://mlsanity.com/paper/2304.13689/) (2023)
- [Speech Retrieval-Augmented Generation without Automatic Speech Recognition](https://mlsanity.com/paper/2412.16500/) (2024)
- [Detecting Extraneous Content in Podcasts](https://mlsanity.com/paper/2103.02585/) (2021)
- [T5lephone: Bridging Speech and Text Self-supervised Models for Spoken Language Understanding via Phoneme level T5](https://mlsanity.com/paper/2211.00586/) (2022)
- [Using Pause Information for More Accurate Entity Recognition](https://mlsanity.com/paper/2109.13222/) (2021)
- [Teach me with a Whisper: Enhancing Large Language Models for Analyzing Spoken Transcripts using Speech Embeddings](https://mlsanity.com/paper/2311.07014/) (2023)
- [Deep Insights into Cognitive Decline: A Survey of Leveraging Non-Intrusive Modalities with Deep Learning Techniques](https://mlsanity.com/paper/2410.18972/) (2024)

## Abstract (arXiv)

While various end-to-end models for spoken language understanding tasks have been explored recently, this paper is probably the first known attempt to challenge the very difficult task of end-to-end spoken question answering (SQA). Learning from the very successful BERT model for various text processing tasks, here we proposed an audio-and-text jointly learned SpeechBERT model. This model outperformed the conventional approach of cascading ASR with the following text question answering (TQA) model on datasets including ASR errors in answer spans, because the end-to-end model was shown to be able to extract information out of audio data before ASR produced errors. When ensembling the proposed end-to-end model with the cascade architecture, even better performance was achieved. In addition to the potential of end-to-end SQA, the SpeechBERT can also be considered for many other spoken language understanding tasks just as BERT for many text processing tasks.