# SD-QA: Spoken Dialectal Question Answering for the Real World

arXiv: https://arxiv.org/abs/2109.12072 · HTML: https://mlsanity.com/paper/2109.12072/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 40 references that are in the map.
- First posted on arXiv 24 Sep 2021.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [QuAC : Question Answering in Context](https://mlsanity.com/paper/1808.07036/) (2018)
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019)
- [Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets](https://mlsanity.com/paper/1908.07898/) (2019)
- [MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering](https://mlsanity.com/paper/2007.15207/) (2020)
- [XOR QA: Cross-lingual Open-Retrieval Question Answering](https://mlsanity.com/paper/2010.11856/) (2020)
- [A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual & Group Unfairness via Inequality Indices](https://mlsanity.com/paper/1807.00787/) (2018)
- [Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehension](https://mlsanity.com/paper/1804.00320/) (2018)
- [Towards Data Distillation for End-to-end Spoken Conversational Question Answering](https://mlsanity.com/paper/2010.08923/) (2020)
- [A BERT Baseline for the Natural Questions](https://mlsanity.com/paper/1901.08634/) (2019)
- [NoiseQA: Challenge Set Evaluation for User-Centric Question Answering](https://mlsanity.com/paper/2102.08345/) (2021)
- [ODSQA: Open-domain Spoken Question Answering Dataset](https://mlsanity.com/paper/1808.02280/) (2018)
- [Multilingual Transfer Learning for QA Using Translation as Data Augmentation](https://mlsanity.com/paper/2012.05958/) (2020)
- [Towards More Equitable Question Answering Systems: How Much More Data Do You Need?](https://mlsanity.com/paper/2105.14115/) (2021)

## Built on by (most cited first, 30 of 38)

- [VoiceBench: Benchmarking LLM-Based Voice Assistants](https://mlsanity.com/paper/2410.17196/) (2024)
- [Systematic Inequalities in Language Technology Performance across the World's Languages](https://mlsanity.com/paper/2110.06733/) (2021)
- [Multi-VALUE: A Framework for Cross-Dialectal English NLP](https://mlsanity.com/paper/2212.08011/) (2022)
- [DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languages](https://mlsanity.com/paper/2403.11009/) (2024)
- [Distilling an End-to-End Voice Assistant Without Instruction Training Data](https://mlsanity.com/paper/2410.02678/) (2024)
- [Dataset Geography: Mapping Language Data to Language Users](https://mlsanity.com/paper/2112.03497/) (2021)
- [HeySQuAD: A Spoken Question Answering Dataset](https://mlsanity.com/paper/2304.13689/) (2023)
- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMs](https://mlsanity.com/paper/2603.24596/) (2026)
- [DUAL: Discrete Spoken Unit Adaptive Learning for Textless Spoken Question Answering](https://mlsanity.com/paper/2203.04911/) (2022)
- [Dialect-robust Evaluation of Generated Text](https://mlsanity.com/paper/2211.00922/) (2022)
- [UNO-Bench: A Unified Benchmark for Exploring the Compositional Law Between Uni-modal and Omni-modal in Omni Models](https://mlsanity.com/paper/2510.18915/) (2025)
- [KenSwQuAD -- A Question Answering Dataset for Swahili Low Resource Language](https://mlsanity.com/paper/2205.02364/) (2022)
- [Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varieties](https://mlsanity.com/paper/2411.10954/) (2024)
- [GlobalBench: A Benchmark for Global Progress in Natural Language Processing](https://mlsanity.com/paper/2305.14716/) (2023)
- [Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Tasks](https://mlsanity.com/paper/2509.20045/) (2025)
- [Mind the Gap! Static and Interactive Evaluations of Large Audio Models](https://mlsanity.com/paper/2502.15919/) (2025)
- [On the Impact of Speech Recognition Errors in Passage Retrieval for Spoken Question Answering](https://mlsanity.com/paper/2209.12944/) (2022)
- [The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoning](https://mlsanity.com/paper/2603.17837/) (2026)
- [CCFQA: A Benchmark for Cross-Lingual and Cross-Modal Speech and Text Factuality Evaluation](https://mlsanity.com/paper/2508.07295/) (2025)
- [Speechless: Speech Instruction Training Without Speech for Low Resource Languages](https://mlsanity.com/paper/2505.17417/) (2025)
- [Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties](https://mlsanity.com/paper/2505.20875/) (2025)
- [VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents](https://mlsanity.com/paper/2510.11098/) (2025)
- [Audio Interaction Model](https://mlsanity.com/paper/2606.05121/) (2026)
- [Dim Wihl Gat Tun: The Case for Linguistic Expertise in NLP for Underdocumented Languages](https://mlsanity.com/paper/2203.09632/) (2022)
- [A Multimodal Dense Retrieval Approach for Speech-Based Open-Domain Question Answering](https://mlsanity.com/paper/2409.13483/) (2024)
- [Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialects](https://mlsanity.com/paper/2510.07890/) (2025)
- [MIST: Multimodal Interactive Speech-based Tool-calling Conversational Assistants for Smart Homes](https://mlsanity.com/paper/2605.06897/) (2026)
- [Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation](https://mlsanity.com/paper/2605.12034/) (2026)
- [Information Asymmetry across Language Varieties: A Case Study on Cantonese-Mandarin and Bavarian-German QA](https://mlsanity.com/paper/2603.14782/) (2026)

## Abstract (arXiv)

Question answering (QA) systems are now available through numerous commercial applications for a wide variety of domains, serving millions of users that interact with them via speech interfaces. However, current benchmarks in QA research do not account for the errors that speech recognition models might introduce, nor do they consider the language variations (dialects) of the users. To address this gap, we augment an existing QA dataset to construct a multi-dialect, spoken QA benchmark on five languages (Arabic, Bengali, English, Kiswahili, Korean) with more than 68k audio prompts in 24 dialects from 255 speakers. We provide baseline results showcasing the real-world performance of QA systems and analyze the effect of language variety and other sensitive speaker attributes on downstream performance. Last, we study the fairness of the ASR and QA models with respect to the underlying user populations. The dataset, model outputs, and code for reproducing all our experiments are available: https://github.com/ffaisal93/SD-QA.