# SpeechDPR: End-to-End Spoken Passage Retrieval for Open-Domain Spoken Question Answering

arXiv: https://arxiv.org/abs/2401.13463 · HTML: https://mlsanity.com/paper/2401.13463/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 24 Jan 2024.

## Builds on (most cited first, 20 of 20)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [End-to-End Open-Domain Question Answering with BERTserini](https://mlsanity.com/paper/1902.01718/) (2019)
- [Direct speech-to-speech translation with discrete units](https://mlsanity.com/paper/2107.05604/) (2021)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehension](https://mlsanity.com/paper/1804.00320/) (2018)
- [Towards End-to-end Unsupervised Speech Recognition](https://mlsanity.com/paper/2204.02492/) (2022)
- [SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks](https://mlsanity.com/paper/2212.10525/) (2022)
- [Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech](https://mlsanity.com/paper/1804.03052/) (2018)
- [ODSQA: Open-domain Spoken Question Answering Dataset](https://mlsanity.com/paper/1808.02280/) (2018)
- [Semantic speech retrieval with a visually grounded model of untranscribed speech](https://mlsanity.com/paper/1710.01949/) (2017)
- [Neural Network based End-to-End Query by Example Spoken Term Detection](https://mlsanity.com/paper/1911.08332/) (2019)
- [Query-by-example Spoken Term Detection using Attention-based Multi-hop Networks](https://mlsanity.com/paper/1709.00354/) (2017)

## Built on by (most cited first, 12 of 12)

- [Speech Retrieval-Augmented Generation without Automatic Speech Recognition](https://mlsanity.com/paper/2412.16500/) (2024)
- [Retrieval-Enhanced Machine Learning: Synthesis and Opportunities](https://mlsanity.com/paper/2407.12982/) (2024)
- [Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approach](https://mlsanity.com/paper/2409.09009/) (2024)
- [VoxRAG: A Step Toward Transcription-Free RAG Systems in Spoken Question Answering](https://mlsanity.com/paper/2505.17326/) (2025)
- [SQuTR: A Robustness Benchmark for Spoken Query to Text Retrieval under Acoustic Noise](https://mlsanity.com/paper/2602.12783/) (2026)
- [From Awareness to Adherence: Bridging the Context Gap in Spoken Dialogue Systems via Context-Aware Decoding](https://mlsanity.com/paper/2606.16472/) (2026)
- [Rhapsody: A Dataset for Highlight Detection in Podcasts](https://mlsanity.com/paper/2505.19429/) (2025)
- [Expressive Speech Retrieval using Natural Language Descriptions of Speaking Style](https://mlsanity.com/paper/2508.11187/) (2025)
- [LuxSQA: Ask Me in Luxembourgish with TTS-Augmented Spoken Question Answering](https://mlsanity.com/paper/2607.02763/) (2026)
- [INSPIRE: A Benchmark for Instruction-Aware Speech Retrieval](https://mlsanity.com/paper/2608.16203/) (2026)
- [Don't Just Listen, Try Planning: Graph-based Retrieval-Generation Agent for Long-form Audio Meeting Understanding](https://mlsanity.com/paper/2608.24048/) (2026)
- [VoiceTrace: A Benchmark and Retrieval Framework for Who-Said-What Speech Retrieval](https://mlsanity.com/paper/2609.18521/) (2026)

## Abstract (arXiv)

Spoken Question Answering (SQA) is essential for machines to reply to user's question by finding the answer span within a given spoken passage. SQA has been previously achieved without ASR to avoid recognition errors and Out-of-Vocabulary (OOV) problems. However, the real-world problem of Open-domain SQA (openSQA), in which the machine needs to first retrieve passages that possibly contain the answer from a spoken archive in addition, was never considered. This paper proposes the first known end-to-end framework, Speech Dense Passage Retriever (SpeechDPR), for the retrieval component of the openSQA problem. SpeechDPR learns a sentence-level semantic representation by distilling knowledge from the cascading model of unsupervised ASR (UASR) and text dense retriever (TDR). No manually transcribed speech data is needed. Initial experiments showed performance comparable to the cascading model of UASR and TDR, and significantly better when UASR was poor, verifying this approach is more robust to speech recognition errors.