mlsanityData to 7 Oct 2026

SD-QA: Spoken Dialectal Question Answering for the Real World

arXiv 2109.12072 ↗First posted 24 Sep 2021cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

Question answering (QA) systems are now available through numerous commercial applications for a wide variety of domains, serving millions of users that interact with them via speech interfaces. However, current benchmarks in QA research do not account for the errors that speech recognition models might introduce, nor do they consider the language variations (dialects) of the users. To address this gap, we augment an existing QA dataset to construct a multi-dialect, spoken QA benchmark on five languages (Arabic, Bengali, English, Kiswahili, Korean) with more than 68k audio prompts in 24 dialects from 255 speakers. We provide baseline results showcasing the real-world performance of QA systems and analyze the effect of language variety and other sensitive speaker attributes on downstream performance. Last, we study the fairness of the ASR and QA models with respect to the underlying user populations. The dataset, model outputs, and code for reproducing all our experiments are available: https://github.com/ffaisal93/SD-QA.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 19 of its 40 references that are in the map. First posted on arXiv 24 Sep 2021.

Builds on

19 of 40 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  5. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  6. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  7. 2018QuAC : Question Answering in Contextcited by 379
  8. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  9. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 173
  10. 2020MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answeringcited by 151
  11. 2020XOR QA: Cross-lingual Open-Retrieval Question Answeringcited by 121
  12. 2018A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual & Group Unfairness via Inequality Indicescited by 97
7 more
  1. 2018Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehensioncited by 74
  2. 2020Towards Data Distillation for End-to-end Spoken Conversational Question Answeringcited by 34
  3. 2019A BERT Baseline for the Natural Questionscited by 33
  4. 2021NoiseQA: Challenge Set Evaluation for User-Centric Question Answeringcited by 25
  5. 2018ODSQA: Open-domain Spoken Question Answering Datasetcited by 21
  6. 2020Multilingual Transfer Learning for QA Using Translation as Data Augmentationcited by 20
  7. 2021Towards More Equitable Question Answering Systems: How Much More Data Do You Need?cited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 7; 2023: 2; 2024: 5; 2025: 9; 2026: 11.

36 papers.

Sort:

By area:

  1. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  2. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 118
  3. 2022Multi-VALUE: A Framework for Cross-Dialectal English NLPcited by 51
  4. 2024DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languagescited by 49
  5. 2024Distilling an End-to-End Voice Assistant Without Instruction Training Datacited by 35
  6. 2021Dataset Geography: Mapping Language Data to Language Userscited by 22
  7. 2023HeySQuAD: A Spoken Question Answering Datasetcited by 16
  8. 2022DUAL: Discrete Spoken Unit Adaptive Learning for Textless Spoken Question Answeringcited by 15
  9. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 15
  10. 2022Dialect-robust Evaluation of Generated Textcited by 14
  11. 2026X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMscited by 13
  12. 2022KenSwQuAD -- A Question Answering Dataset for Swahili Low Resource Languagecited by 12
24 more
  1. 2025UNO-Bench: A Unified Benchmark for Exploring the Compositional Law Between Uni-modal and Omni-modal in Omni Modelscited by 12
  2. 2024Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varietiescited by 10
  3. 2025Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Taskscited by 9
  4. 2023GlobalBench: A Benchmark for Global Progress in Natural Language Processingcited by 8
  5. 2025Mind the Gap! Static and Interactive Evaluations of Large Audio Modelscited by 8
  6. 2022On the Impact of Speech Recognition Errors in Passage Retrieval for Spoken Question Answeringcited by 6
  7. 2025CCFQA: A Benchmark for Cross-Lingual and Cross-Modal Speech and Text Factuality Evaluationcited by 5
  8. 2026The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoningcited by 5
  9. 2025Speechless: Speech Instruction Training Without Speech for Low Resource Languagescited by 4
  10. 2025Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varietiescited by 4
  11. 2025VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agentscited by 4
  12. 2026Audio Interaction Modelcited by 3
  13. 2022Dim Wihl Gat Tun: The Case for Linguistic Expertise in NLP for Underdocumented Languagescited by 2
  14. 2024A Multimodal Dense Retrieval Approach for Speech-Based Open-Domain Question Answeringcited by 2
  15. 2025Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialectscited by 2
  16. 2026MIST: Multimodal Interactive Speech-based Tool-calling Conversational Assistants for Smart Homescited by 2
  17. 2026Information Asymmetry across Language Varieties: A Case Study on Cantonese-Mandarin and Bavarian-German QAcited by 1
  18. 2026AuRA: Internalizing Audio Understanding into LLMs as LoRAcited by 1
  19. 2022Towards leveraging latent knowledge and Dialogue context for real-world conversational question answeringcited by 0
  20. 2026Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluationcited by 0
  21. 2026KVoiceBench, KOpenAudioBench, and KMMAU: Agent-Driven Korean Speech Benchmarks for Evaluating SpeechLMscited by 0
  22. 2026Better Retrieval, Worse Robustness: How Multi-hop RAG Amplifies Upstream ASR Errorscited by 0
  23. 2026Nuha-Speech: Building General-Purpose Arabic Speech-LLMscited by 0
  24. 2026V=akQA: A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answeringcited by 0