# Is Semantic Chunking Worth the Computational Cost?

arXiv: https://arxiv.org/abs/2410.13070 · HTML: https://mlsanity.com/paper/2410.13070/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Placed in

- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 17 of 17)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models](https://mlsanity.com/paper/2311.09210/) (2023)
- [RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2407.11005/) (2024)
- [DelucionQA: Detecting Hallucinations in Domain-specific Question Answering](https://mlsanity.com/paper/2312.05200/) (2023)
- [The Chronicles of RAG: The Retriever, the Chunk and the Generator](https://mlsanity.com/paper/2401.07883/) (2024)
- [ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers](https://mlsanity.com/paper/2110.06884/) (2021)
- [From RAGs to rich parameters: Probing how language models utilize external knowledge over parametric information for factual queries](https://mlsanity.com/paper/2406.12824/) (2024)

## Built on by (most cited first, 30 of 43)

- [LightMem: Lightweight and Efficient Memory-Augmented Generation](https://mlsanity.com/paper/2510.18866/) (2025)
- [MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented Generation System](https://mlsanity.com/paper/2503.09600/) (2025)
- [Towards Reliable Retrieval in RAG Systems for Large Legal Datasets](https://mlsanity.com/paper/2510.06999/) (2025)
- [Classifying and Addressing the Diversity of Errors in Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2510.13975/) (2025)
- [Chain-of-Memory: Lightweight Memory Construction with Dynamic Evolution for LLM Agents](https://mlsanity.com/paper/2601.14287/) (2026)
- [MultiDocFusion: Hierarchical and Multimodal Chunking Pipeline for Enhanced RAG on Long Industrial Documents](https://mlsanity.com/paper/2604.12352/) (2026)
- [SemRAG: Semantic Knowledge-Augmented RAG for Improved Question-Answering](https://mlsanity.com/paper/2507.21110/) (2025)
- [SafeRBench: Dissecting the Reasoning Safety of Large Language Models](https://mlsanity.com/paper/2511.15169/) (2025)
- [Taxonomy of the Retrieval System Framework: Pitfalls and Paradigms](https://mlsanity.com/paper/2601.20131/) (2026)
- [TOBUGraph: Knowledge Graph-Based Retrieval for Enhanced LLM Performance Beyond RAG](https://mlsanity.com/paper/2412.05447/) (2024)
- [Practical Code RAG at Scale: Task-Aware Retrieval Design Choices under Compute Budgets](https://mlsanity.com/paper/2510.20609/) (2025)
- [A Systematic Investigation of Document Chunking Strategies and Embedding Sensitivity](https://mlsanity.com/paper/2603.06976/) (2026)
- [HiKEY: Hierarchical Multimodal Retrieval for Open-Domain Document Question Answering](https://mlsanity.com/paper/2605.29606/) (2026)
- [Multi-Agent System for AI-Assisted Extraction of Narrative Arcs in TV Series](https://mlsanity.com/paper/2503.04817/) (2025)
- [Can LLMs Replace Humans During Code Chunking?](https://mlsanity.com/paper/2506.19897/) (2025)
- [An Agile Method for Implementing Retrieval Augmented Generation Tools in Industrial SMEs](https://mlsanity.com/paper/2508.21024/) (2025)
- [Paper Circle: An Open-source Multi-agent Research Discovery and Analysis Framework](https://mlsanity.com/paper/2604.06170/) (2026)
- [Beyond Text: Characterizing Domain Expert Needs in Document Research](https://mlsanity.com/paper/2504.12495/) (2025)
- [Less Context, Same Performance: A RAG Framework for Resource-Efficient LLM-Based Clinical NLP](https://mlsanity.com/paper/2505.20320/) (2025)
- [Knowledge Compression via Question Generation: Enhancing Multihop Document Retrieval without Fine-tuning](https://mlsanity.com/paper/2506.13778/) (2025)
- [MoM: Mixtures of Scenario-Aware Document Memories for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2510.14252/) (2025)
- [Disco-RAG: Discourse-Aware Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.04377/) (2026)
- [Semantic Chunking and the Entropy of Natural Language](https://mlsanity.com/paper/2602.13194/) (2026)
- [Evaluating Chunking Strategies For Retrieval-Augmented Generation in Oil and Gas Enterprise Documents](https://mlsanity.com/paper/2603.24556/) (2026)
- [Adaptive Chunking: Optimizing Chunking-Method Selection for RAG](https://mlsanity.com/paper/2603.25333/) (2026)
- [AnchorMem: Anchored Facts with Associative Contexts for Building Memory in Large Language Models](https://mlsanity.com/paper/2604.17377/) (2026)
- [Chunking Methods on Retrieval-Augmented Generation - Effectiveness Evaluation Against Computational Cost and Limitations](https://mlsanity.com/paper/2606.00881/) (2026)
- [GRASP: GRanularity-Aware Search Policy for Agentic RAG](https://mlsanity.com/paper/2607.10463/) (2026)
- [Embedding-Based Context-Aware Reranker](https://mlsanity.com/paper/2510.13329/) (2025)
- [FreeChunker: A Cross-Granularity Chunking Framework](https://mlsanity.com/paper/2510.20356/) (2025)

## Abstract (arXiv)

Recent advances in Retrieval-Augmented Generation (RAG) systems have popularized semantic chunking, which aims to improve retrieval performance by dividing documents into semantically coherent segments. Despite its growing adoption, the actual benefits over simpler fixed-size chunking, where documents are split into consecutive, fixed-size segments, remain unclear. This study systematically evaluates the effectiveness of semantic chunking using three common retrieval-related tasks: document retrieval, evidence retrieval, and retrieval-based answer generation. The results show that the computational costs associated with semantic chunking are not justified by consistent performance gains. These findings challenge the previous assumptions about semantic chunking and highlight the need for more efficient chunking strategies in RAG systems.