# Recurrent Chunking Mechanisms for Long-Text Machine Reading Comprehension

arXiv: https://arxiv.org/abs/2005.08056 · HTML: https://mlsanity.com/paper/2005.08056/

## Key facts

- Cited by 29 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 16 May 2020.

## Built on by (most cited first, 29 of 29)

- [Retrieval-Augmented Generation for Natural Language Processing: A Survey](https://mlsanity.com/paper/2407.13193/) (2024)
- [BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models](https://mlsanity.com/paper/2309.13345/) (2023)
- [A Survey on Long Text Modeling with Transformers](https://mlsanity.com/paper/2302.14502/) (2023)
- [End-to-end Spoken Conversational Question Answering: Task, Dataset and Model](https://mlsanity.com/paper/2204.14272/) (2022)
- [RoR: Read-over-Read for Long Document Machine Reading Comprehension](https://mlsanity.com/paper/2109.04780/) (2021)
- [Specialized Document Embeddings for Aspect-based Similarity of Research Papers](https://mlsanity.com/paper/2203.14541/) (2022)
- [Text Matching Improves Sequential Recommendation by Reducing Popularity Biases](https://mlsanity.com/paper/2308.14029/) (2023)
- [Capturing Global Structural Information in Long Document Question Answering with Compressive Graph Selector Network](https://mlsanity.com/paper/2210.05499/) (2022)
- [Chunk, Align, Select: A Simple Long-sequence Processing Method for Transformers](https://mlsanity.com/paper/2308.13191/) (2023)
- [LogPr'ecis: Unleashing Language Models for Automated Malicious Log Analysis](https://mlsanity.com/paper/2307.08309/) (2023)
- [Drilling Down into the Discourse Structure with LLMs for Long Document Question Answering](https://mlsanity.com/paper/2311.13565/) (2023)
- [InfoAgent: Advancing Autonomous Information-Seeking Agents](https://mlsanity.com/paper/2509.25189/) (2025)
- [SPAR: Personalized Content-Based Recommendation via Long Engagement Attention](https://mlsanity.com/paper/2402.10555/) (2024)
- [An Evaluation of Large Language Models on Text Summarization Tasks Using Prompt Engineering Techniques](https://mlsanity.com/paper/2507.05123/) (2025)
- [MultiDocFusion: Hierarchical and Multimodal Chunking Pipeline for Enhanced RAG on Long Industrial Documents](https://mlsanity.com/paper/2604.12352/) (2026)
- [AspectCSE: Sentence Embeddings for Aspect-based Semantic Textual Similarity Using Contrastive Learning and Structured Knowledge](https://mlsanity.com/paper/2307.07851/) (2023)
- [The Law of Large Documents: Understanding the Structure of Legal Contracts Using Visual Cues](https://mlsanity.com/paper/2107.08128/) (2021)
- [Dynamic Sliding Window for Meeting Summarization](https://mlsanity.com/paper/2108.13629/) (2021)
- [Long-context Reference-based MT Quality Estimation](https://mlsanity.com/paper/2509.13980/) (2025)
- [HiKEY: Hierarchical Multimodal Retrieval for Open-Domain Document Question Answering](https://mlsanity.com/paper/2605.29606/) (2026)
- [FETILDA: An Effective Framework For Fin-tuned Embeddings For Long Financial Text Documents](https://mlsanity.com/paper/2206.06952/) (2022)
- [Knowledge Compression via Question Generation: Enhancing Multihop Document Retrieval without Fine-tuning](https://mlsanity.com/paper/2506.13778/) (2025)
- [EnSI-RAG: Entity-Structure-Indexed Retrieval-Augmented Generation for Long-Document Question Answering](https://mlsanity.com/paper/2608.21252/) (2026)
- [VAULT: VAriable Unified Long Text Representation for Machine Reading Comprehension](https://mlsanity.com/paper/2105.03229/) (2021)
- [Deep Understanding based Multi-Document Machine Reading Comprehension](https://mlsanity.com/paper/2204.03494/) (2022)
- [An Understanding-Oriented Robust Machine Reading Comprehension Model](https://mlsanity.com/paper/2207.00187/) (2022)
- [PyZoBot: A Platform for Conversational Information Extraction and Synthesis from Curated Zotero Reference Libraries through Advanced Retrieval-Augmented Generation](https://mlsanity.com/paper/2405.07963/) (2024)
- [Personality Prediction from Life Stories using Language Models](https://mlsanity.com/paper/2506.19258/) (2025)
- [M3DocDep: Multi-modal, Multi-page, Multi-document Dependency Chunking with Large Vision-Language Models](https://mlsanity.com/paper/2605.18774/) (2026)

## Abstract (arXiv)

In this paper, we study machine reading comprehension (MRC) on long texts, where a model takes as inputs a lengthy document and a question and then extracts a text span from the document as an answer. State-of-the-art models tend to use a pretrained transformer model (e.g., BERT) to encode the joint contextual information of document and question. However, these transformer-based models can only take a fixed-length (e.g., 512) text as its input. To deal with even longer text inputs, previous approaches usually chunk them into equally-spaced segments and predict answers based on each segment independently without considering the information from other segments. As a result, they may form segments that fail to cover the correct answer span or retain insufficient contexts around it, which significantly degrades the performance. Moreover, they are less capable of answering questions that need cross-segment information. We propose to let a model learn to chunk in a more flexible way via reinforcement learning: a model can decide the next segment that it wants to process in either direction. We also employ recurrent mechanisms to enable information to flow across segments. Experiments on three MRC datasets -- CoQA, QuAC, and TriviaQA -- demonstrate the effectiveness of our proposed recurrent chunking mechanisms: we can obtain segments that are more likely to contain complete answers and at the same time provide sufficient contexts around the ground truth answers for better predictions.