# Attention-over-Attention Neural Networks for Reading Comprehension

arXiv: https://arxiv.org/abs/1607.04423 · HTML: https://mlsanity.com/paper/1607.04423/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jul 2016.

## Built on by (most cited first, 30 of 31)

- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Biomedical Question Answering: A Survey of Approaches and Challenges](https://mlsanity.com/paper/2102.05281/) (2021)
- [Explain and Predict, and then Predict Again](https://mlsanity.com/paper/2101.04109/) (2021)
- [Learning Fine-grained Fact-Article Correspondence in Legal Cases](https://mlsanity.com/paper/2104.10726/) (2021)
- [Less Learn Shortcut: Analyzing and Mitigating Learning of Spurious Feature-Label Correlation](https://mlsanity.com/paper/2205.12593/) (2022)
- [Neural Quality Estimation with Multiple Hypotheses for Grammatical Error Correction](https://mlsanity.com/paper/2105.04443/) (2021)
- [Enhance Long Text Understanding via Distilled Gist Detector from Abstractive Summarization](https://mlsanity.com/paper/2110.04741/) (2021)
- [Coarse-grained decomposition and fine-grained interaction for multi-hop question answering](https://mlsanity.com/paper/2101.05988/) (2021)
- [Multilingual Multi-Aspect Explainability Analyses on Machine Reading Comprehension Models](https://mlsanity.com/paper/2108.11574/) (2021)
- [ExpMRC: Explainability Evaluation for Machine Reading Comprehension](https://mlsanity.com/paper/2105.04126/) (2021)
- [Fact-driven Logical Reasoning for Machine Reading Comprehension](https://mlsanity.com/paper/2105.10334/) (2021)
- [Contextual embedding and model weighting by fusing domain knowledge on Biomedical Question Answering](https://mlsanity.com/paper/2206.12866/) (2022)
- [Sparse Attentive Memory Network for Click-through Rate Prediction with Long Sequences](https://mlsanity.com/paper/2208.04022/) (2022)
- [A Neural Model for Joint Document and Snippet Ranking in Question Answering for Large Document Collections](https://mlsanity.com/paper/2106.08908/) (2021)
- [To Answer or Not to Answer? Improving Machine Reading Comprehension Model with Span-based Contrastive Learning](https://mlsanity.com/paper/2208.01299/) (2022)
- [Optimized Biomedical Question-Answering Services with LLM and Multi-BERT Integration](https://mlsanity.com/paper/2410.12856/) (2024)
- [ClueReader: Heterogeneous Graph Attention Network for Multi-hop Machine Reading Comprehension](https://mlsanity.com/paper/2107.00841/) (2021)
- [Denoised Non-Local Neural Network for Semantic Segmentation](https://mlsanity.com/paper/2110.14200/) (2021)
- [A Comprehensive Survey on Multi-hop Machine Reading Comprehension Datasets and Metrics](https://mlsanity.com/paper/2212.04070/) (2022)
- [A Comprehensive Survey on Multi-hop Machine Reading Comprehension Approaches](https://mlsanity.com/paper/2212.04072/) (2022)
- [Orthogonal Attention: A Cloze-Style Approach to Negation Scope Resolution](https://mlsanity.com/paper/2103.04294/) (2021)
- [Subsentence Extraction from Text Using Coverage-Based Deep Learning Language Models](https://mlsanity.com/paper/2104.09777/) (2021)
- [VAULT: VAriable Unified Long Text Representation for Machine Reading Comprehension](https://mlsanity.com/paper/2105.03229/) (2021)
- [Deep Understanding based Multi-Document Machine Reading Comprehension](https://mlsanity.com/paper/2204.03494/) (2022)
- [NFLAT: Non-Flat-Lattice Transformer for Chinese Named Entity Recognition](https://mlsanity.com/paper/2205.05832/) (2022)
- [An Understanding-Oriented Robust Machine Reading Comprehension Model](https://mlsanity.com/paper/2207.00187/) (2022)
- [Modular Approach to Machine Reading Comprehension: Mixture of Task-Aware Experts](https://mlsanity.com/paper/2210.01750/) (2022)
- [RCMHA: Relative Convolutional Multi-Head Attention for Natural Language Modelling](https://mlsanity.com/paper/2308.03429/) (2023)
- [Clinical Reading Comprehension with Encoder-Decoder Models Enhanced by Direct Preference Optimization](https://mlsanity.com/paper/2407.14000/) (2024)
- [Improving Prediction of Need for Mechanical Ventilation using Cross-Attention](https://mlsanity.com/paper/2407.15885/) (2024)

## Abstract (arXiv)

Cloze-style queries are representative problems in reading comprehension. Over the past few months, we have seen much progress that utilizing neural network approach to solve Cloze-style questions. In this paper, we present a novel model called attention-over-attention reader for the Cloze-style reading comprehension task. Our model aims to place another attention mechanism over the document-level attention, and induces "attended attention" for final predictions. Unlike the previous works, our neural network model requires less pre-defined hyper-parameters and uses an elegant architecture for modeling. Experimental results show that the proposed attention-over-attention model significantly outperforms various state-of-the-art systems by a large margin in public datasets, such as CNN and Children's Book Test datasets.