# RoR: Read-over-Read for Long Document Machine Reading Comprehension

arXiv: https://arxiv.org/abs/2109.04780 · HTML: https://mlsanity.com/paper/2109.04780/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 30 references that are in the map.
- First posted on arXiv 10 Sep 2021.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [QuAC : Question Answering in Context](https://mlsanity.com/paper/1808.07036/) (2018)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine](https://mlsanity.com/paper/1704.05179/) (2017)
- [Neural Approaches to Conversational AI](https://mlsanity.com/paper/1809.08267/) (2018)
- [BERT with History Answer Embedding for Conversational Question Answering](https://mlsanity.com/paper/1905.05412/) (2019)
- [RevCore: Review-augmented Conversational Recommendation](https://mlsanity.com/paper/2106.00957/) (2021)
- [GraphFlow: Exploiting Conversation Flow with Graph Neural Networks for Conversational Machine Comprehension](https://mlsanity.com/paper/1908.00059/) (2019)
- [Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching](https://mlsanity.com/paper/2004.12297/) (2020)
- [FlowQA: Grasping Flow in History for Conversational Machine Comprehension](https://mlsanity.com/paper/1810.06683/) (2018)
- [Attentive History Selection for Conversational Question Answering](https://mlsanity.com/paper/1908.09456/) (2019)
- [Recurrent Chunking Mechanisms for Long-Text Machine Reading Comprehension](https://mlsanity.com/paper/2005.08056/) (2020)
- [FlowDelta: Modeling Flow Information Gain in Reasoning for Conversational Machine Comprehension](https://mlsanity.com/paper/1908.05117/) (2019)
- [Multi-Passage Machine Reading Comprehension with Cross-Passage Answer Verification](https://mlsanity.com/paper/1805.02220/) (2018)
- [Conversational Machine Comprehension: a Literature Review](https://mlsanity.com/paper/2006.00671/) (2020)

## Built on by (most cited first, 15 of 15)

- [Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactions](https://mlsanity.com/paper/2312.02913/) (2023)
- [A Survey on Long Text Modeling with Transformers](https://mlsanity.com/paper/2302.14502/) (2023)
- [Saving Dense Retriever from Shortcut Dependency in Conversational Search](https://mlsanity.com/paper/2202.07280/) (2022)
- [Generating Information-Seeking Conversations from Unlabeled Documents](https://mlsanity.com/paper/2205.12609/) (2022)
- [On the Robustness of Dialogue History Representation in Conversational Question Answering: A Comprehensive Study and a New Prompt-based Method](https://mlsanity.com/paper/2206.14796/) (2022)
- [MuGER2: Multi-Granularity Evidence Retrieval and Reasoning for Hybrid Question Answering](https://mlsanity.com/paper/2210.10350/) (2022)
- [Can Question Rewriting Help Conversational Question Answering?](https://mlsanity.com/paper/2204.06239/) (2022)
- [CoHS-CQG: Context and History Selection for Conversational Question Generation](https://mlsanity.com/paper/2209.06652/) (2022)
- [OPERA:Operation-Pivoted Discrete Reasoning over Text](https://mlsanity.com/paper/2204.14166/) (2022)
- [CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling](https://mlsanity.com/paper/2406.12018/) (2024)
- [S2M: Converting Single-Turn to Multi-Turn Datasets for Conversational Question Answering](https://mlsanity.com/paper/2312.16511/) (2023)
- [ChuLo: Chunk-Level Key Information Representation for Long Document Understanding](https://mlsanity.com/paper/2410.11119/) (2024)
- [Logically at Factify 2: A Multi-Modal Fact Checking System Based on Evidence Retrieval techniques and Transformer Encoder Architecture](https://mlsanity.com/paper/2301.03127/) (2023)
- [Trustworthy Large Models in Vision: A Survey](https://mlsanity.com/paper/2311.09680/) (2023)
- [Hierarchical Multi-modal Transformer for Cross-modal Long Document Classification](https://mlsanity.com/paper/2407.10105/) (2024)

## Abstract (arXiv)

Transformer-based pre-trained models, such as BERT, have achieved remarkable results on machine reading comprehension. However, due to the constraint of encoding length (e.g., 512 WordPiece tokens), a long document is usually split into multiple chunks that are independently read. It results in the reading field being limited to individual chunks without information collaboration for long document machine reading comprehension. To address this problem, we propose RoR, a read-over-read method, which expands the reading field from chunk to document. Specifically, RoR includes a chunk reader and a document reader. The former first predicts a set of regional answers for each chunk, which are then compacted into a highly-condensed version of the original document, guaranteeing to be encoded once. The latter further predicts the global answers from this condensed document. Eventually, a voting strategy is utilized to aggregate and rerank the regional and global answers for final prediction. Extensive experiments on two benchmarks QuAC and TriviaQA demonstrate the effectiveness of RoR for long document reading. Notably, RoR ranks 1st place on the QuAC leaderboard (https://quac.ai/) at the time of submission (May 17th, 2021).