# Speech-Based Visual Question Answering

arXiv: https://arxiv.org/abs/1705.00464 · HTML: https://mlsanity.com/paper/1705.00464/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 May 2017.

## Built on by (most cited first, 10 of 10)

- [Learning to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2203.14072/) (2022)
- [Progressive Spatio-temporal Perception for Audio-Visual Question Answering](https://mlsanity.com/paper/2308.05421/) (2023)
- [TVLT: Textless Vision-Language Transformer](https://mlsanity.com/paper/2209.14156/) (2022)
- [Multi-Scale Attention for Audio Question Answering](https://mlsanity.com/paper/2305.17993/) (2023)
- [Recent, rapid advancement in visual question answering architecture: a review](https://mlsanity.com/paper/2203.01322/) (2022)
- [NAAQA: A Neural Architecture for Acoustic Question Answering](https://mlsanity.com/paper/2106.06147/) (2021)
- [Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunities](https://mlsanity.com/paper/2210.16960/) (2022)
- [TM-PATHVQA:90000+ Textless Multilingual Questions for Medical Visual Question Answering](https://mlsanity.com/paper/2407.11383/) (2024)
- [Spoken question answering for visual queries](https://mlsanity.com/paper/2505.23308/) (2025)
- [SViQA: A Unified Speech-Vision Multimodal Model for Textless Visual Question Answering](https://mlsanity.com/paper/2504.01049/) (2025)

## Abstract (arXiv)

This paper introduces speech-based visual question answering (VQA), the task of generating an answer given an image and a spoken question. Two methods are studied: an end-to-end, deep neural network that directly uses audio waveforms as input versus a pipelined approach that performs ASR (Automatic Speech Recognition) on the question, followed by text-based visual question answering. Furthermore, we investigate the robustness of both methods by injecting various levels of noise into the spoken question and find both methods to be tolerate noise at similar levels.