# A Systematic Investigation of Document Chunking Strategies and Embedding Sensitivity

arXiv: https://arxiv.org/abs/2603.06976 · HTML: https://mlsanity.com/paper/2603.06976/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 33 of its 47 references that are in the map.
- First posted on arXiv 7 Mar 2026.

## Placed in

- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 30 of 33)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers](https://mlsanity.com/paper/2012.15828/) (2020)
- [Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering](https://mlsanity.com/paper/2502.06193/) (2025)
- [Financial Report Chunking for Effective Retrieval Augmented Generation](https://mlsanity.com/paper/2402.05131/) (2024)
- [Is Semantic Chunking Worth the Computational Cost?](https://mlsanity.com/paper/2410.13070/) (2024)
- [Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement](https://mlsanity.com/paper/2510.09738/) (2025)
- [Meta-Chunking: Learning Text Segmentation and Semantic Completion via Logical Perception](https://mlsanity.com/paper/2410.12788/) (2024)
- [Rethinking Chunk Size For Long-Document Retrieval: A Multi-Dataset Analysis](https://mlsanity.com/paper/2505.21700/) (2025)
- [Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges](https://mlsanity.com/paper/2504.15205/) (2025)
- [Reconstructing Context: Evaluating Advanced Chunking Strategies for Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.19754/) (2025)
- [HiChunk: Evaluating and Enhancing Retrieval-Augmented Generation with Hierarchical Chunking](https://mlsanity.com/paper/2509.11552/) (2025)
- [Towards Reliable Retrieval in RAG Systems for Large Legal Datasets](https://mlsanity.com/paper/2510.06999/) (2025)
- [AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agents](https://mlsanity.com/paper/2512.23343/) (2025)
- [Benchmarking Retrieval-Augmented Large Language Models in Biomedical NLP: Application, Robustness, and Self-Awareness](https://mlsanity.com/paper/2405.08151/) (2024)
- [MultiFinRAG: An Optimized Multimodal Retrieval-Augmented Generation (RAG) Framework for Financial Question Answering](https://mlsanity.com/paper/2506.20821/) (2025)
- [Enhancing Retrieval Augmented Generation with Hierarchical Text Segmentation Chunking](https://mlsanity.com/paper/2507.09935/) (2025)
- [A New HOPE: Domain-agnostic Automatic Evaluation of Text Chunking](https://mlsanity.com/paper/2505.02171/) (2025)
- [Passage Segmentation of Documents for Extractive Question Answering](https://mlsanity.com/paper/2501.09940/) (2025)
- [Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.17277/) (2025)
- [RAGSmith: A Framework for Finding the Optimal Composition of Retrieval-Augmented Generation Methods Across Datasets](https://mlsanity.com/paper/2511.01386/) (2025)
- [RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generation](https://mlsanity.com/paper/2508.06056/) (2025)
- [Metadata-Driven Retrieval-Augmented Generation for Financial Question Answering](https://mlsanity.com/paper/2510.24402/) (2025)

## Built on by (most cited first, 3 of 3)

- [FinTradeBench: A Financial Reasoning Benchmark for LLMs](https://mlsanity.com/paper/2603.19225/) (2026)
- [Structure-Aware Chunking for Tabular Data in Retrieval-Augmented Generation](https://mlsanity.com/paper/2605.00318/) (2026)
- [Does Document Structure Help Dense Retrieval? A Placebo-Controlled Ablation of Four Mechanisms Across Two Corpora](https://mlsanity.com/paper/2610.10170/) (2026)

## Abstract (arXiv)

We present the first large-scale, cross-domain evaluation of document chunking strategies for dense retrieval, addressing a critical but underexplored aspect of retrieval-augmented systems. In our study, 36 segmentation methods spanning fixed-size, semantic, structure-aware, hierarchical, adaptive, and LLM-assisted approaches are benchmarked across six diverse knowledge domains using five different embedding models. Retrieval performance is assessed using graded relevance scores from a state-of-the-art LLM evaluator, with Normalised DCG@5 as the primary metric (complemented by Hit@5 and MRR). Our experiments show that content-aware chunking significantly improves retrieval effectiveness over naive fixed-length splitting. The top-performing strategy, Paragraph Group Chunking, achieved the highest overall accuracy (mean nDCG@5~0.459) and substantially better top-rank hit rates (Precision@1~24%, Hit@5~59%). In contrast, simple fixed-size character chunking as baselines performed poorly (nDCG@5 < 0.244, Precision@1~2-3%). We observe pronounced domain-specific differences: dynamic token sizing is strongest in biology, physics and health, while paragraph grouping is strongest in legal and maths. Larger embedding models yield higher absolute scores but remain sensitive to suboptimal segmentation, indicating that better chunking and large embeddings provide complementary benefits. In addition to accuracy gains, we quantify the efficiency trade-offs of advanced chunking. Producing more, smaller chunks can increase index size and latency. Consequently, we identify methods (like dynamic chunking) that approach an optimal balance of effectiveness and efficiency. These findings establish chunking as a vital lever for improving retrieval performance and reliability.