# LumberChunker: Long-Form Narrative Document Segmentation

arXiv: https://arxiv.org/abs/2406.17526 · HTML: https://mlsanity.com/paper/2406.17526/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 23 references that are in the map.
- First posted on arXiv 25 Jun 2024.

## Builds on (most cited first, 11 of 11)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [Precise Zero-Shot Dense Retrieval without Relevance Labels](https://mlsanity.com/paper/2212.10496/) (2022)
- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024)
- [Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models](https://mlsanity.com/paper/2311.09210/) (2023)
- [Pre-trained Language Models Can be Fully Zero-Shot Learners](https://mlsanity.com/paper/2212.06950/) (2022)

## Built on by (most cited first, 30 of 36)

- [LightMem: Lightweight and Efficient Memory-Augmented Generation](https://mlsanity.com/paper/2510.18866/) (2025)
- [On Memory Construction and Retrieval for Personalized Conversational Agents](https://mlsanity.com/paper/2502.05589/) (2025)
- [A Survey on Knowledge-Oriented Retrieval-Augmented Generation](https://mlsanity.com/paper/2503.10677/) (2025)
- [Meta-Chunking: Learning Text Segmentation and Semantic Completion via Logical Perception](https://mlsanity.com/paper/2410.12788/) (2024)
- [UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generation](https://mlsanity.com/paper/2410.02719/) (2024)
- [RAG Meets Temporal Graphs: Time-Sensitive Modeling and Retrieval for Evolving Knowledge](https://mlsanity.com/paper/2510.13590/) (2025)
- [HiChunk: Evaluating and Enhancing Retrieval-Augmented Generation with Hierarchical Chunking](https://mlsanity.com/paper/2509.11552/) (2025)
- [Introducing a new hyper-parameter for RAG: Context Window Utilization](https://mlsanity.com/paper/2407.19794/) (2024)
- [Passage Segmentation of Documents for Extractive Question Answering](https://mlsanity.com/paper/2501.09940/) (2025)
- [Chain-of-Memory: Lightweight Memory Construction with Dynamic Evolution for LLM Agents](https://mlsanity.com/paper/2601.14287/) (2026)
- [Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaboration](https://mlsanity.com/paper/2505.21471/) (2025)
- [HiKEY: Hierarchical Multimodal Retrieval for Open-Domain Document Question Answering](https://mlsanity.com/paper/2605.29606/) (2026)
- [Can LLMs Replace Humans During Code Chunking?](https://mlsanity.com/paper/2506.19897/) (2025)
- [Chronological Passage Assembling in RAG framework for Temporal Question Answering](https://mlsanity.com/paper/2508.18748/) (2025)
- [PAKTON: A Multi-Agent Framework for Question Answering in Long Legal Agreements](https://mlsanity.com/paper/2506.00608/) (2025)
- [Knowledge Compression via Question Generation: Enhancing Multihop Document Retrieval without Fine-tuning](https://mlsanity.com/paper/2506.13778/) (2025)
- [Leveraging LLMs to Streamline the Review of Public Funding Applications](https://mlsanity.com/paper/2510.09674/) (2025)
- [MoM: Mixtures of Scenario-Aware Document Memories for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2510.14252/) (2025)
- [Semantic Chunking and the Entropy of Natural Language](https://mlsanity.com/paper/2602.13194/) (2026)
- [Adaptive Chunking: Optimizing Chunking-Method Selection for RAG](https://mlsanity.com/paper/2603.25333/) (2026)
- [AnchorMem: Anchored Facts with Associative Contexts for Building Memory in Large Language Models](https://mlsanity.com/paper/2604.17377/) (2026)
- [A Benchmark Construction and Evaluation Framework for Specialist Domains: Case Study on Defense-related Documents](https://mlsanity.com/paper/2604.17943/) (2026)
- [Chunking Methods on Retrieval-Augmented Generation - Effectiveness Evaluation Against Computational Cost and Limitations](https://mlsanity.com/paper/2606.00881/) (2026)
- [Right Reset: Chunking by Prefix Removal](https://mlsanity.com/paper/2608.04330/) (2026)
- [Semantic Source Code Segmentation using Small and Large Language Models](https://mlsanity.com/paper/2507.08992/) (2025)
- [From Evidence-Based Medicine to Knowledge Graph: Retrieval-Augmented Generation for Sports Rehabilitation and a Domain Benchmark](https://mlsanity.com/paper/2601.00216/) (2026)
- [SegNSP: Revisiting Next Sentence Prediction for Linear Text Segmentation](https://mlsanity.com/paper/2601.03474/) (2026)
- [M'as contexto no es mejor. Paradoja de la diluci'on vectorial en RAG corporativos](https://mlsanity.com/paper/2601.08851/) (2025)
- [RegGuard: AI-Powered Retrieval-Enhanced Assistant for Pharmaceutical Regulatory Compliance](https://mlsanity.com/paper/2601.17826/) (2026)
- [TopoChunker: Topology-Aware Agentic Document Chunking Framework](https://mlsanity.com/paper/2603.18409/) (2026)

## Abstract (arXiv)

Modern NLP tasks increasingly rely on dense retrieval methods to access up-to-date and relevant contextual information. We are motivated by the premise that retrieval benefits from segments that can vary in size such that a content's semantic independence is better captured. We propose LumberChunker, a method leveraging an LLM to dynamically segment documents, which iteratively prompts the LLM to identify the point within a group of sequential passages where the content begins to shift. To evaluate our method, we introduce GutenQA, a benchmark with 3000 "needle in a haystack" type of question-answer pairs derived from 100 public domain narrative books available on Project Gutenberg. Our experiments show that LumberChunker not only outperforms the most competitive baseline by 7.37% in retrieval performance (DCG@20) but also that, when integrated into a RAG pipeline, LumberChunker proves to be more effective than other chunking methods and competitive baselines, such as the Gemini 1.5M Pro. Our Code and Data are available at https://github.com/joaodsmarques/LumberChunker