# AiR: Attention with Reasoning Capability

arXiv: https://arxiv.org/abs/2007.14419 · HTML: https://mlsanity.com/paper/2007.14419/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jul 2020.

## Built on by (most cited first, 28 of 28)

- [Grounding Answers for Visual Questions Asked by Visually Impaired People](https://mlsanity.com/paper/2202.01993/) (2022)
- [Going Beyond XAI: A Systematic Survey for Explanation-Guided Learning](https://mlsanity.com/paper/2212.03954/) (2022)
- [Found a Reason for me? Weakly-supervised Grounded Visual Question Answering using Capsules](https://mlsanity.com/paper/2105.04836/) (2021)
- [RES: A Robust Framework for Guiding Visual Explanation](https://mlsanity.com/paper/2206.13413/) (2022)
- [Multimodal Integration of Human-Like Attention in Visual Question Answering](https://mlsanity.com/paper/2109.13139/) (2021)
- [On Expressivity and Trainability of Quadratic Networks](https://mlsanity.com/paper/2110.06081/) (2021)
- [Learning from Observer Gaze:Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognition](https://mlsanity.com/paper/2405.09931/) (2024)
- [VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering](https://mlsanity.com/paper/2109.13116/) (2021)
- [GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpaths](https://mlsanity.com/paper/2408.02788/) (2024)
- [VQA Therapy: Exploring Answer Differences by Visually Grounding Answers](https://mlsanity.com/paper/2308.11662/) (2023)
- [REX: Reasoning-aware and Grounded Explanation](https://mlsanity.com/paper/2203.06107/) (2022)
- [Beyond Average: Individualized Visual Scanpath Prediction](https://mlsanity.com/paper/2404.12235/) (2024)
- [Where, What, Why: Towards Explainable Driver Attention Prediction](https://mlsanity.com/paper/2506.23088/) (2025)
- [Trends, Applications, and Challenges in Human Attention Modelling](https://mlsanity.com/paper/2402.18673/) (2024)
- [Look Hear: Gaze Prediction for Speech-directed Human Attention](https://mlsanity.com/paper/2407.19605/) (2024)
- [DiffEye: Diffusion-Based Continuous Eye-Tracking Data Generation Conditioned on Natural Images](https://mlsanity.com/paper/2509.16767/) (2025)
- [Contrastive Language-Image Pretrained Models are Zero-Shot Human Scanpath Predictors](https://mlsanity.com/paper/2305.12380/) (2023)
- [Visual Attention Prompted Prediction and Learning](https://mlsanity.com/paper/2310.08420/) (2023)
- [DiffGaze: A Diffusion Model for Continuous Gaze Sequence Generation on 360deg Images](https://mlsanity.com/paper/2403.17477/) (2024)
- [Attention Alignment Between Humans and Vision-Language Models](https://mlsanity.com/paper/2606.17410/) (2026)
- [Attention in Reasoning: Dataset, Analysis, and Modeling](https://mlsanity.com/paper/2204.09774/) (2022)
- [GazeSearch: Radiology Findings Search Benchmark](https://mlsanity.com/paper/2411.05780/) (2024)
- [MagiC: Evaluating Multimodal Cognition Toward Grounded Visual Reasoning](https://mlsanity.com/paper/2507.07297/) (2025)
- [MoCA: Incorporating Multi-stage Domain Pretraining and Cross-guided Multimodal Attention for Textbook Question Answering](https://mlsanity.com/paper/2112.02839/) (2021)
- [Predicting Visual Attention and Distraction During Visual Search Using Convolutional Neural Networks](https://mlsanity.com/paper/2210.15093/) (2022)
- [A Framework for Generating Artificial Datasets to Validate Absolute and Relative Position Concepts](https://mlsanity.com/paper/2509.18177/) (2025)
- [TDSal: Task-Based Top-Down Saliency Prediction Model](https://mlsanity.com/paper/2607.09827/) (2026)
- [Gaze-to-text Generation: Beyond Categorical Decoding of Human Attention](https://mlsanity.com/paper/2607.23917/) (2026)

## Abstract (arXiv)

While attention has been an increasingly popular component in deep neural networks to both interpret and boost performance of models, little work has examined how attention progresses to accomplish a task and whether it is reasonable. In this work, we propose an Attention with Reasoning capability (AiR) framework that uses attention to understand and improve the process leading to task outcomes. We first define an evaluation metric based on a sequence of atomic reasoning operations, enabling quantitative measurement of attention that considers the reasoning process. We then collect human eye-tracking and answer correctness data, and analyze various machine and human attentions on their reasoning capability and how they impact task performance. Furthermore, we propose a supervision method to jointly and progressively optimize attention, reasoning, and task performance so that models learn to look at regions of interests by following a reasoning process. We demonstrate the effectiveness of the proposed framework in analyzing and modeling attention with better reasoning capability and task performance. The code and data are available at https://github.com/szzexpoi/AiR