# DUAL: Discrete Spoken Unit Adaptive Learning for Textless Spoken Question Answering

arXiv: https://arxiv.org/abs/2203.04911 · HTML: https://mlsanity.com/paper/2203.04911/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 9 Mar 2022.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [Pretrained Transformers as Universal Computation Engines](https://mlsanity.com/paper/2103.05247/) (2021)
- [Direct speech-to-speech translation with discrete units](https://mlsanity.com/paper/2107.05604/) (2021)
- [Textless Speech-to-Speech Translation on Real Data](https://mlsanity.com/paper/2112.08352/) (2021)
- [Text-Free Prosody-Aware Generative Spoken Language Modeling](https://mlsanity.com/paper/2109.03264/) (2021)
- [Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehension](https://mlsanity.com/paper/1804.00320/) (2018)
- [SD-QA: Spoken Dialectal Question Answering for the Real World](https://mlsanity.com/paper/2109.12072/) (2021)
- [Towards Data Distillation for End-to-end Spoken Conversational Question Answering](https://mlsanity.com/paper/2010.08923/) (2020)
- [NoiseQA: Challenge Set Evaluation for User-Centric Question Answering](https://mlsanity.com/paper/2102.08345/) (2021)
- [On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets](https://mlsanity.com/paper/2109.03537/) (2021)
- [Towards Machine Comprehension of Spoken Content: Initial TOEFL Listening Comprehension Test by Machine](https://mlsanity.com/paper/1608.06378/) (2016)
- [ODSQA: Open-domain Spoken Question Answering Dataset](https://mlsanity.com/paper/1808.02280/) (2018)
- [Mitigating the Impact of Speech Recognition Errors on Spoken Question Answering by Adversarial Domain Adaptation](https://mlsanity.com/paper/1904.07904/) (2019)
- [Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models' Transferability](https://mlsanity.com/paper/2103.07162/) (2021)

## Built on by (most cited first, 15 of 15)

- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities](https://mlsanity.com/paper/2503.04721/) (2025)
- [Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback](https://mlsanity.com/paper/2411.01834/) (2024)
- [GSQA: An End-to-End Model for Generative Spoken Question Answering](https://mlsanity.com/paper/2312.09781/) (2023)
- [Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks](https://mlsanity.com/paper/2401.02921/) (2024)
- [SpokenNativQA: Multilingual Everyday Spoken Queries for LLMs](https://mlsanity.com/paper/2505.19163/) (2025)
- [Improving Textless Spoken Language Understanding with Discrete Units as Intermediate Target](https://mlsanity.com/paper/2305.18096/) (2023)
- [The Role of Prosody in Spoken Question Answering](https://mlsanity.com/paper/2502.05389/) (2025)
- [The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation](https://mlsanity.com/paper/2305.09652/) (2023)
- [Introducing Semantics into Speech Encoders](https://mlsanity.com/paper/2211.08402/) (2022)
- [Textless Speech-to-Speech Translation With Limited Parallel Data](https://mlsanity.com/paper/2305.15405/) (2023)
- [VoxRAG: A Step Toward Transcription-Free RAG Systems in Spoken Question Answering](https://mlsanity.com/paper/2505.17326/) (2025)
- [EMO-Reasoning: Benchmarking Emotional Reasoning Capabilities in Spoken Dialogue Systems](https://mlsanity.com/paper/2508.17623/) (2025)
- [Task Oriented Dialogue as a Catalyst for Self-Supervised Automatic Speech Recognition](https://mlsanity.com/paper/2401.02417/) (2024)
- [INSPIRE: A Benchmark for Instruction-Aware Speech Retrieval](https://mlsanity.com/paper/2608.16203/) (2026)

## Abstract (arXiv)

Spoken Question Answering (SQA) is to find the answer from a spoken document given a question, which is crucial for personal assistants when replying to the queries from the users. Existing SQA methods all rely on Automatic Speech Recognition (ASR) transcripts. Not only does ASR need to be trained with massive annotated data that are time and cost-prohibitive to collect for low-resourced languages, but more importantly, very often the answers to the questions include name entities or out-of-vocabulary words that cannot be recognized correctly. Also, ASR aims to minimize recognition errors equally over all words, including many function words irrelevant to the SQA task. Therefore, SQA without ASR transcripts (textless) is always highly desired, although known to be very difficult. This work proposes Discrete Spoken Unit Adaptive Learning (DUAL), leveraging unlabeled data for pre-training and fine-tuned by the SQA downstream task. The time intervals of spoken answers can be directly predicted from spoken documents. We also release a new SQA benchmark corpus, NMSQA, for data with more realistic scenarios. We empirically showed that DUAL yields results comparable to those obtained by cascading ASR and text QA model and robust to real-world data. Our code and model will be open-sourced.