# Intent-Driven Dynamic Chunking: Segmenting Documents to Reflect Predicted Information Needs

arXiv: https://arxiv.org/abs/2602.14784 · HTML: https://mlsanity.com/paper/2602.14784/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 11 references that are in the map.
- First posted on arXiv 16 Feb 2026.

## Builds on (most cited first, 5 of 5)

- [Promptagator: Few-shot Dense Retrieval From 8 Examples](https://mlsanity.com/paper/2209.11755/) (2022)
- [Document Expansion by Query Prediction](https://mlsanity.com/paper/1904.08375/) (2019)
- [Text Segmentation as a Supervised Learning Task](https://mlsanity.com/paper/1803.09337/) (2018)
- [InPars: Data Augmentation for Information Retrieval using Large Language Models](https://mlsanity.com/paper/2202.05144/) (2022)
- [Recent Trends in Linear Text Segmentation: a Survey](https://mlsanity.com/paper/2411.16613/) (2024)

## Abstract (arXiv)

Breaking long documents into smaller segments is a fundamental challenge in information retrieval. Whether for search engines, question-answering systems, or retrieval-augmented generation (RAG), effective segmentation determines how well systems can locate and return relevant information. However, traditional methods, such as fixed-length or coherence-based segmentation, ignore user intent, leading to chunks that split answers or contain irrelevant noise. We introduce Intent-Driven Dynamic Chunking (IDC), a novel approach that uses predicted user queries to guide document segmentation. IDC leverages a Large Language Model to generate likely user intents for a document and then employs a dynamic programming algorithm to find the globally optimal chunk boundaries. This represents a novel application of DP to intent-aware segmentation that avoids greedy pitfalls. We evaluated IDC on six diverse question-answering datasets, including news articles, Wikipedia, academic papers, and technical documentation. IDC outperformed traditional chunking strategies on five datasets, improving top-1 retrieval accuracy by 5% to 67%, and matched the best baseline on the sixth. Additionally, IDC produced 40-60% fewer chunks than baseline methods while achieving 93-100% answer coverage. These results demonstrate that aligning document structure with anticipated information needs significantly boosts retrieval performance, particularly for long and heterogeneous documents.