# Rethinking Chunk Size For Long-Document Retrieval: A Multi-Dataset Analysis

arXiv: https://arxiv.org/abs/2505.21700 · HTML: https://mlsanity.com/paper/2505.21700/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 27 May 2025.

## Placed in

- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [End-to-End Training of Neural Retrievers for Open-Domain Question Answering](https://mlsanity.com/paper/2101.00408/) (2021)
- [Dense Hierarchical Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2110.15439/) (2021)
- [The TechQA Dataset](https://mlsanity.com/paper/1911.02984/) (2019)
- [Mix-of-Granularity: Optimize the Chunking Granularity for Retrieval-Augmented Generation](https://mlsanity.com/paper/2406.00456/) (2024)

## Built on by (most cited first, 17 of 17)

- [HiChunk: Evaluating and Enhancing Retrieval-Augmented Generation with Hierarchical Chunking](https://mlsanity.com/paper/2509.11552/) (2025)
- [ContextPilot: Fast Long-Context Inference via Context Reuse](https://mlsanity.com/paper/2511.03475/) (2025)
- [Dynamic Context Adaptation for Consistent Role-Playing Agents with Retrieval-Augmented Generations](https://mlsanity.com/paper/2508.02016/) (2025)
- [A Systematic Investigation of Document Chunking Strategies and Embedding Sensitivity](https://mlsanity.com/paper/2603.06976/) (2026)
- [An Agile Method for Implementing Retrieval Augmented Generation Tools in Industrial SMEs](https://mlsanity.com/paper/2508.21024/) (2025)
- [Knowledge-Grounded Agentic Large Language Models for Multi-Hazard Understanding from Reconnaissance Reports](https://mlsanity.com/paper/2511.14010/) (2025)
- [Modeling Language as a Sequence of Thoughts](https://mlsanity.com/paper/2512.25026/) (2025)
- [SmartChunk Retrieval: Query-Aware Chunk Compression with Planning for Efficient Document RAG](https://mlsanity.com/paper/2602.22225/) (2025)
- [The Effect of Text Chunk Size on Retrieval-Augmented Generation Performance](https://mlsanity.com/paper/2607.24767/) (2026)
- [RAG Strategies for Natural Language-Based SQL Query and REST API Call Generation](https://mlsanity.com/paper/2602.07086/) (2026)
- [QChunker: Learning Question-Aware Text Chunking for Domain RAG via Multi-Agent Debate](https://mlsanity.com/paper/2603.11650/) (2026)
- [Domain-Adapted Retrieval for In-Context Annotation of Pedagogical Dialogue Acts](https://mlsanity.com/paper/2604.03127/) (2026)
- [MMClima: A Framework for Multimodal Climate Science Data and Evaluation](https://mlsanity.com/paper/2606.10194/) (2026)
- [Lost in a Single Vector: Improving Long-Document Retrieval with Chunk Evidence Aggregation](https://mlsanity.com/paper/2606.18781/) (2026)
- [Cross-Attention Calibrated Deduplication for Retrieval-Augmented Generation System](https://mlsanity.com/paper/2607.24332/) (2026)
- [Structure-Aware Semantic Chunking with Title-Chain Prefixes: A 1600-Query Evaluation and the Measurement Trap in Text-Transform Ablations](https://mlsanity.com/paper/2608.00824/) (2026)
- [Does Document Structure Help Dense Retrieval? A Placebo-Controlled Ablation of Four Mechanisms Across Two Corpora](https://mlsanity.com/paper/2610.10170/) (2026)

## Abstract (arXiv)

Chunking is a crucial preprocessing step in retrieval-augmented generation (RAG) systems, significantly impacting retrieval effectiveness across diverse datasets. In this study, we systematically evaluate fixed-size chunking strategies and their influence on retrieval performance using multiple embedding models. Our experiments, conducted on both short-form and long-form datasets, reveal that chunk size plays a critical role in retrieval effectiveness -- smaller chunks (64-128 tokens) are optimal for datasets with concise, fact-based answers, whereas larger chunks (512-1024 tokens) improve retrieval in datasets requiring broader contextual understanding. We also analyze the impact of chunking on different embedding models, finding that they exhibit distinct chunking sensitivities. While models like Stella benefit from larger chunks, leveraging global context for long-range retrieval, Snowflake performs better with smaller chunks, excelling at fine-grained, entity-based matching. Our results underscore the trade-offs between chunk size, embedding models, and dataset characteristics, emphasizing the need for improved chunk quality measures, and more comprehensive datasets to advance chunk-based retrieval in long-document Information Retrieval (IR).