mlsanityData to 14 Oct 2026

ReadTwice: Reading Very Large Documents with Memories

arXiv 2105.04241 ↗First posted 10 May 2021cs.CL cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Knowledge-intensive tasks such as question answering often require assimilating information from different sections of large inputs such as books or article collections. We propose ReadTwice, a simple and effective technique that combines several strengths of prior approaches to model long-range dependencies with Transformers. The main idea is to read text in small segments, in parallel, summarizing each segment into a memory table to be used in a second read of the text. We show that the method outperforms models of comparable size on several question answering (QA) datasets and sets a new state of the art on the challenging NarrativeQA task, with questions about entire books. Source code and pre-trained checkpoints for ReadTwice can be found at https://goo.gle/research-readtwice.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 24 of its 28 references that are in the map. First posted on arXiv 10 May 2021.

Builds on

24 of 28 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2020Longformer: The Long-Document Transformercited by 3,413
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  10. 2018Self-Attention with Relative Position Representationscited by 1,120
  11. 2020Efficient Transformers: A Surveycited by 790
  12. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
12 more
  1. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  2. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  3. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 302
  4. 2019Knowledge Enhanced Contextual Word Representationscited by 296
  5. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  6. 2019HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarizationcited by 121
  7. 2017Simple and Effective Multi-Paragraph Reading Comprehensioncited by 120
  8. 2020Entities as Experts: Sparse Memory Access with Entity Supervisioncited by 100
  9. 2017Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generationcited by 61
  10. 2020Probabilistic Assumptions Matter: Improved Models for Distantly-Supervised Document-Level Question Answeringcited by 17
  11. 2019Language Model Pre-training for Hierarchical Document Representationscited by 7
  12. 2020Frustratingly Hard Evidence Retrieval for QA Over Bookscited by 7

Built on by

Cited by year

Citing papers per year: 2021: 4; 2022: 4; 2023: 6.

14 papers.

Sort:

By area:

  1. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  2. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  3. 2022Memorizing Transformerscited by 226
  4. 2022Training Language Models with Memory Augmentationcited by 113
  5. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  6. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  7. 2023Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manualscited by 45
  8. 2021Mention Memory: incorporating textual knowledge into Transformers through entity mention attentioncited by 43
  9. 2022FiDO: Fusion-in-Decoder optimized for stronger performance and faster inferencecited by 31
  10. 2023Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your computecited by 17
  11. 2021Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Studycited by 16
  12. 2023MEMORY-VQ: Compression for Tractable Internet-Scale Memorycited by 2
2 more
  1. 2023Uncertainty Guided Global Memory Improves Multi-Hop Question Answeringcited by 2
  2. 2022Deep Understanding based Multi-Document Machine Reading Comprehensioncited by 0