mlsanityData to 14 Oct 2026

Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering

arXiv 2204.09634 ↗First posted 20 Apr 2022cs.SD cs.LG eess.AScited by 85

Not on the map. Why?

Abstract (arXiv)

Audio question answering (AQA) is a multimodal translation task where a system analyzes an audio signal and a natural language question, to generate a desirable natural language answer. In this paper, we introduce Clotho-AQA, a dataset for Audio question answering consisting of 1991 audio files each between 15 to 30 seconds in duration selected from the Clotho dataset. For each audio file, we collect six different questions and corresponding answers by crowdsourcing using Amazon Mechanical Turk. The questions and answers are produced by different annotators. Out of the six questions for each audio, two questions each are designed to have 'yes' and 'no' as answers, while the remaining two questions have other single-word answers. For each question, we collect answers from three different annotators. We also present two baseline experiments to describe the usage of our dataset for the AQA task - an LSTM-based multimodal binary classifier for 'yes' or 'no' type answers and an LSTM-based multimodal multi-class classifier for 828 single-word answers. The binary classifier achieved an accuracy of 62.7% and the multi-class classifier achieved a top-1 accuracy of 54.2% and a top-5 accuracy of 93.7%. Clotho-AQA dataset is freely available online at https://zenodo.org/record/6473207.

Key facts. Cited by 85 papers in the map (data through 14 Oct 2026). Builds on 19 of its 24 references that are in the map. First posted on arXiv 20 Apr 2022.

Builds on

19 of 24 references.

References · 19

  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  2. 2015VQA: Visual Question Answeringcited by 2,506
  3. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  4. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  5. 2017Look, Listen and Learncited by 458
  6. 2019Clotho: An Audio Captioning Datasetcited by 358
  7. 2016NewsQA: A Machine Comprehension Datasetcited by 331
  8. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  9. 2017Advances in Pre-Training Distributed Word Representationscited by 207
  10. 2014A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Inputcited by 112
  11. 2015Yin and Yang: Balancing and Answering Binary Visual Questionscited by 109
  12. 2015Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answeringcited by 80
7 more
  1. 2017DeepStory: Video Story QA by Deep Embedded Memory Networkscited by 68
  2. 2016Visual Question Answering: Datasets, Algorithms, and Future Challengescited by 56
  3. 2019Temporal Reasoning via Audio Question Answeringcited by 39
  4. 2015Simple Baseline for Visual Question Answeringcited by 24
  5. 2015Exploring Nearest Neighbor Approaches for Image Captioningcited by 19
  6. 2021Recent Advances in Video Question Answering: A Review of Datasets and Methodscited by 12
  7. 2018CLEAR: A Dataset for Compositional Language and Elementary Acoustic Reasoningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 22; 2025: 35; 2026: 20.

85 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  2. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  3. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
  4. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267
  5. 2025Kimi-Audio Technical Reportcited by 222
  6. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
  7. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
  8. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  9. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilitiescited by 156
  10. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 133
  11. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 128
  12. 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 117
38 more
  1. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 108
  2. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 80
  3. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  4. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 59
  5. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 39
  6. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 36
  7. 2025URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Modelscited by 30
  8. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 29
  9. 2025LongCat-Flash-Omni Technical Reportcited by 26
  10. 2025Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decodingcited by 24
  11. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  12. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 23
  13. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  14. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
  15. 2025Learning Musical Representations for Music Performance Question Answeringcited by 17
  16. 2025The iNaturalist Sounds Datasetcited by 17
  17. 2023Multi-Scale Attention for Audio Question Answeringcited by 13
  18. 2025OmniEval: A Benchmark for Evaluating Omni-modal Models with Visual, Auditory, and Textual Inputscited by 13
  19. 2025When Audio and Text Disagree: Revealing Text Bias in Large Audio-Language Modelscited by 13
  20. 2024LLMs Can Evolve Continually on Modality for X-Modal Reasoningcited by 12
  21. 2025Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samplescited by 11
  22. 2026ERNIE 5.0 Technical Reportcited by 11
  23. 2023Investigating the Emergent Audio Classification Ability of ASR Foundation Modelscited by 10
  24. 2025TAMP: Token-Adaptive Layerwise Pruning in Multimodal Large Language Modelscited by 9
  25. 2025TACOS: Temporally-aligned Audio CaptiOnS for Language-Audio Pretrainingcited by 9
  26. 2024CoMMIT: Coordinated Multimodal Instruction Tuningcited by 8
  27. 2024Audio Dialogues: Dialogues dataset for audio and music understandingcited by 7
  28. 2024SHMamba: Structured Hyperbolic State Space Model for Audio-Visual Question Answeringcited by 7
  29. 2025From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Datacited by 7
  30. 2025StressTest: Can YOUR Speech LM Handle the Stress?cited by 7
  31. 2025SageLM: A Multi-aspect and Explainable Large Language Model for Speech Judgementcited by 7
  32. 2025ARGUS: Defending Against Multimodal Indirect Prompt Injection via Steering Instruction-Following Behaviorcited by 7
  33. 2026VoxPrivacy: A Benchmark for Evaluating Interactional Privacy of Speech Language Modelscited by 7
  34. 2026Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesiscited by 7
  35. 2024Modality-Inconsistent Continual Learning of Multimodal Large Language Modelscited by 6
  36. 2025Merge then Realign: Simple and Effective Modality-Incremental Continual Learning for Multimodal LLMscited by 6
  37. 2025MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Taskscited by 6
  38. 2024Explore the Limits of Omni-modal Pretraining at Scalecited by 5

50 most cited of 85 All 85 (JSON)