mlsanityData to 7 Oct 2026

LumberChunker: Long-Form Narrative Document Segmentation

arXiv 2406.17526 ↗First posted 25 Jun 2024cs.CL cs.IRcited by 36

Not on the map. Why?

Abstract (arXiv)

Modern NLP tasks increasingly rely on dense retrieval methods to access up-to-date and relevant contextual information. We are motivated by the premise that retrieval benefits from segments that can vary in size such that a content's semantic independence is better captured. We propose LumberChunker, a method leveraging an LLM to dynamically segment documents, which iteratively prompts the LLM to identify the point within a group of sequential passages where the content begins to shift. To evaluate our method, we introduce GutenQA, a benchmark with 3000 "needle in a haystack" type of question-answer pairs derived from 100 public domain narrative books available on Project Gutenberg. Our experiments show that LumberChunker not only outperforms the most competitive baseline by 7.37% in retrieval performance (DCG@20) but also that, when integrated into a RAG pipeline, LumberChunker proves to be more effective than other chunking methods and competitive baselines, such as the Gemini 1.5M Pro. Our Code and Data are available at https://github.com/joaodsmarques/LumberChunker

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 11 of its 23 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

11 of 23 references.

References · 11

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  5. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  6. 2022Large Language Models Can Self-Improvecited by 656
  7. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  8. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  9. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  10. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Modelscited by 180
  11. 2022Pre-trained Language Models Can be Fully Zero-Shot Learnerscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 15; 2026: 18.

36 papers.

Sort:

By area:

  1. 2025LightMem: Lightweight and Efficient Memory-Augmented Generationcited by 163
  2. 2025On Memory Construction and Retrieval for Personalized Conversational Agentscited by 101
  3. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 34
  4. 2024Meta-Chunking: Learning Text Segmentation and Semantic Completion via Logical Perceptioncited by 17
  5. 2024UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generationcited by 14
  6. 2025RAG Meets Temporal Graphs: Time-Sensitive Modeling and Retrieval for Evolving Knowledgecited by 10
  7. 2025HiChunk: Evaluating and Enhancing Retrieval-Augmented Generation with Hierarchical Chunkingcited by 9
  8. 2024Introducing a new hyper-parameter for RAG: Context Window Utilizationcited by 8
  9. 2025Passage Segmentation of Documents for Extractive Question Answeringcited by 6
  10. 2026Chain-of-Memory: Lightweight Memory Construction with Dynamic Evolution for LLM Agentscited by 5
  11. 2025Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaborationcited by 4
  12. 2025Can LLMs Replace Humans During Code Chunking?cited by 2
24 more
  1. 2025Chronological Passage Assembling in RAG framework for Temporal Question Answeringcited by 2
  2. 2026HiKEY: Hierarchical Multimodal Retrieval for Open-Domain Document Question Answeringcited by 2
  3. 2025PAKTON: A Multi-Agent Framework for Question Answering in Long Legal Agreementscited by 1
  4. 2025Knowledge Compression via Question Generation: Enhancing Multihop Document Retrieval without Fine-tuningcited by 1
  5. 2025Leveraging LLMs to Streamline the Review of Public Funding Applicationscited by 1
  6. 2025MoM: Mixtures of Scenario-Aware Document Memories for Retrieval-Augmented Generation Systemscited by 1
  7. 2026Semantic Chunking and the Entropy of Natural Languagecited by 1
  8. 2026Adaptive Chunking: Optimizing Chunking-Method Selection for RAGcited by 1
  9. 2026A Benchmark Construction and Evaluation Framework for Specialist Domains: Case Study on Defense-related Documentscited by 1
  10. 2026Chunking Methods on Retrieval-Augmented Generation - Effectiveness Evaluation Against Computational Cost and Limitationscited by 1
  11. 2026Right Reset: Chunking by Prefix Removalcited by 1
  12. 2025Semantic Source Code Segmentation using Small and Large Language Modelscited by 0
  13. 2026From Evidence-Based Medicine to Knowledge Graph: Retrieval-Augmented Generation for Sports Rehabilitation and a Domain Benchmarkcited by 0
  14. 2026SegNSP: Revisiting Next Sentence Prediction for Linear Text Segmentationcited by 0
  15. 2025M'as contexto no es mejor. Paradoja de la diluci'on vectorial en RAG corporativoscited by 0
  16. 2026RegGuard: AI-Powered Retrieval-Enhanced Assistant for Pharmaceutical Regulatory Compliancecited by 0
  17. 2026TopoChunker: Topology-Aware Agentic Document Chunking Frameworkcited by 0
  18. 2026MDKeyChunker: Single-Call LLM Enrichment with Rolling Keys and Key-Based Restructuring for High-Accuracy RAGcited by 0
  19. 2026AnchorMem: Anchored Facts with Associative Contexts for Building Memory in Large Language Modelscited by 0
  20. 2026M3DocDep: Multi-modal, Multi-page, Multi-document Dependency Chunking with Large Vision-Language Modelscited by 0
  21. 2026LitSeg: Narrative-Aware Document Segmentation for Literary RAGcited by 0
  22. 2026AutoIndex: Learning Representation Programs for Retrievalcited by 0
  23. 2026Caught in the Story: Narrative Captivity in Multi-turn LLMs Conversationcited by 0
  24. 2026Zero-Shot Cue-Grounded Topic Segmentation of Spoken Documentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Storyline Trees: Hierarchical Representations for Long-Form Narratives
  2. 2026Can Structural Cues Save LLMs? Evaluating Language Models in Massive Document Streams
  3. 2026Intent-Driven Dynamic Chunking: Segmenting Documents to Reflect Predicted Information Needs
  4. 2025Enhancing Retrieval Augmented Generation with Hierarchical Text Segmentation Chunking
  5. 2025FreeChunker: A Cross-Granularity Chunking Framework
  6. 2026A Systematic Investigation of Document Chunking Strategies and Embedding Sensitivity
  7. 2026Comparing Chunking and Embedding Strategies for Turkish RAG Systems
  8. 2026Narrative World Model: Narratology-Grounded Writer Memory for Long-Form Fiction