# Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering

arXiv: https://arxiv.org/abs/2204.09634 · HTML: https://mlsanity.com/paper/2204.09634/

## Key facts

- Cited by 85 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 24 references that are in the map.
- First posted on arXiv 20 Apr 2022.

## Builds on (most cited first, 19 of 19)

- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering](https://mlsanity.com/paper/1906.02467/) (2019)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [Visual7W: Grounded Question Answering in Images](https://mlsanity.com/paper/1511.03416/) (2015)
- [Advances in Pre-Training Distributed Word Representations](https://mlsanity.com/paper/1712.09405/) (2017)
- [A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input](https://mlsanity.com/paper/1410.0210/) (2014)
- [Yin and Yang: Balancing and Answering Binary Visual Questions](https://mlsanity.com/paper/1511.05099/) (2015)
- [Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering](https://mlsanity.com/paper/1505.05612/) (2015)
- [DeepStory: Video Story QA by Deep Embedded Memory Networks](https://mlsanity.com/paper/1707.00836/) (2017)
- [Visual Question Answering: Datasets, Algorithms, and Future Challenges](https://mlsanity.com/paper/1610.01465/) (2016)
- [Temporal Reasoning via Audio Question Answering](https://mlsanity.com/paper/1911.09655/) (2019)
- [Simple Baseline for Visual Question Answering](https://mlsanity.com/paper/1512.02167/) (2015)
- [Exploring Nearest Neighbor Approaches for Image Captioning](https://mlsanity.com/paper/1505.04467/) (2015)
- [Recent Advances in Video Question Answering: A Review of Datasets and Methods](https://mlsanity.com/paper/2101.05954/) (2021)
- [CLEAR: A Dataset for Compositional Language and Elementary Acoustic Reasoning](https://mlsanity.com/paper/1811.10561/) (2018)

## Built on by (most cited first, 30 of 85)

- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [InternVideo2: Scaling Foundation Models for Multimodal Video Understanding](https://mlsanity.com/paper/2403.15377/) (2024)
- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [Kimi-Audio Technical Report](https://mlsanity.com/paper/2504.18425/) (2025)
- [AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension](https://mlsanity.com/paper/2402.07729/) (2024)
- [OneLLM: One Framework to Align All Modalities with Language](https://mlsanity.com/paper/2312.03700/) (2023)
- [Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities](https://mlsanity.com/paper/2402.01831/) (2024)
- [GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities](https://mlsanity.com/paper/2406.11768/) (2024)
- [Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities](https://mlsanity.com/paper/2503.03983/) (2025)
- [AudioBench: A Universal Benchmark for Audio Large Language Models](https://mlsanity.com/paper/2406.16020/) (2024)
- [MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix](https://mlsanity.com/paper/2505.13032/) (2025)
- [Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts](https://mlsanity.com/paper/2405.11273/) (2024)
- [X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning](https://mlsanity.com/paper/2311.18799/) (2023)
- [A Survey of Multimodal Large Language Model from A Data-centric Perspective](https://mlsanity.com/paper/2405.16640/) (2024)
- [OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM](https://mlsanity.com/paper/2510.15870/) (2025)
- [Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2505.15957/) (2025)
- [LongCat-Next: Lexicalizing Modalities as Discrete Tokens](https://mlsanity.com/paper/2603.27538/) (2026)
- [URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models](https://mlsanity.com/paper/2502.17810/) (2025)
- [InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue](https://mlsanity.com/paper/2510.13747/) (2025)
- [LongCat-Flash-Omni Technical Report](https://mlsanity.com/paper/2511.00279/) (2025)
- [Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decoding](https://mlsanity.com/paper/2506.07233/) (2025)
- [Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?](https://mlsanity.com/paper/2402.12025/) (2024)
- [Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs](https://mlsanity.com/paper/2405.16700/) (2024)
- [From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalities](https://mlsanity.com/paper/2412.11694/) (2024)
- [Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data](https://mlsanity.com/paper/2511.12609/) (2025)
- [Learning Musical Representations for Music Performance Question Answering](https://mlsanity.com/paper/2502.06710/) (2025)
- [The iNaturalist Sounds Dataset](https://mlsanity.com/paper/2506.00343/) (2025)
- [Multi-Scale Attention for Audio Question Answering](https://mlsanity.com/paper/2305.17993/) (2023)
- [OmniEval: A Benchmark for Evaluating Omni-modal Models with Visual, Auditory, and Textual Inputs](https://mlsanity.com/paper/2506.20960/) (2025)

## Abstract (arXiv)

Audio question answering (AQA) is a multimodal translation task where a system analyzes an audio signal and a natural language question, to generate a desirable natural language answer. In this paper, we introduce Clotho-AQA, a dataset for Audio question answering consisting of 1991 audio files each between 15 to 30 seconds in duration selected from the Clotho dataset. For each audio file, we collect six different questions and corresponding answers by crowdsourcing using Amazon Mechanical Turk. The questions and answers are produced by different annotators. Out of the six questions for each audio, two questions each are designed to have 'yes' and 'no' as answers, while the remaining two questions have other single-word answers. For each question, we collect answers from three different annotators. We also present two baseline experiments to describe the usage of our dataset for the AQA task - an LSTM-based multimodal binary classifier for 'yes' or 'no' type answers and an LSTM-based multimodal multi-class classifier for 828 single-word answers. The binary classifier achieved an accuracy of 62.7% and the multi-class classifier achieved a top-1 accuracy of 54.2% and a top-5 accuracy of 93.7%. Clotho-AQA dataset is freely available online at https://zenodo.org/record/6473207.