# Beyond VQA: Generating Multi-word Answer and Rationale to Visual Questions

arXiv: https://arxiv.org/abs/2010.12852 · HTML: https://mlsanity.com/paper/2010.12852/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2020.

## Built on by (most cited first, 19 of 19)

- [e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks](https://mlsanity.com/paper/2105.03761/) (2021)
- [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021)
- [HRVQA: A Visual Question Answering Benchmark for High-Resolution Aerial Images](https://mlsanity.com/paper/2301.09460/) (2023)
- [Discovering Spatio-Temporal Rationales for Video Question Answering](https://mlsanity.com/paper/2307.12058/) (2023)
- [Long-Form Answers to Visual Questions from Blind and Low Vision People](https://mlsanity.com/paper/2408.06303/) (2024)
- [GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpaths](https://mlsanity.com/paper/2408.02788/) (2024)
- [VQA and Visual Reasoning: An Overview of Recent Datasets, Methods and Challenges](https://mlsanity.com/paper/2212.13296/) (2022)
- [REX: Reasoning-aware and Grounded Explanation](https://mlsanity.com/paper/2203.06107/) (2022)
- [ChartQA-X: Generating Explanations for Visual Chart Reasoning](https://mlsanity.com/paper/2504.13275/) (2025)
- [Chunk-aware Alignment and Lexical Constraint for Visual Entailment with Natural Language Explanations](https://mlsanity.com/paper/2207.11401/) (2022)
- [Redundancy-aware Transformer for Video Question Answering](https://mlsanity.com/paper/2308.03267/) (2023)
- [ERVQA: A Dataset to Benchmark the Readiness of Large Vision Language Models in Hospital Environments](https://mlsanity.com/paper/2410.06420/) (2024)
- [Towards a Unified Model for Generating Answers and Explanations in Visual Question Answering](https://mlsanity.com/paper/2301.10799/) (2023)
- [On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization](https://mlsanity.com/paper/2205.11686/) (2022)
- [Dynamic Clue Bottlenecks: Towards Interpretable-by-Design Visual Question Answering](https://mlsanity.com/paper/2305.14882/) (2023)
- [A Survey on Interpretable Cross-modal Reasoning](https://mlsanity.com/paper/2309.01955/) (2023)
- [Multimodal Rationales for Explainable Visual Question Answering](https://mlsanity.com/paper/2402.03896/) (2024)
- [Preconditioned Visual Language Inference with Weak Supervision](https://mlsanity.com/paper/2306.01753/) (2023)
- [HLTCOE Evaluation Team at TREC 2025: VQA Track](https://mlsanity.com/paper/2512.07738/) (2025)

## Abstract (arXiv)

Visual Question Answering is a multi-modal task that aims to measure high-level visual understanding. Contemporary VQA models are restrictive in the sense that answers are obtained via classification over a limited vocabulary (in the case of open-ended VQA), or via classification over a set of multiple-choice-type answers. In this work, we present a completely generative formulation where a multi-word answer is generated for a visual query. To take this a step forward, we introduce a new task: ViQAR (Visual Question Answering and Reasoning), wherein a model must generate the complete answer and a rationale that seeks to justify the generated answer. We propose an end-to-end architecture to solve this task and describe how to evaluate it. We show that our model generates strong answers and rationales through qualitative and quantitative evaluation, as well as through a human Turing Test.