# Mix-of-Granularity: Optimize the Chunking Granularity for Retrieval-Augmented Generation

arXiv: https://arxiv.org/abs/2406.00456 · HTML: https://mlsanity.com/paper/2406.00456/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge-graph and structured-knowledge RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 1 Jun 2024.

## Placed in

- Retrieval-augmented generation / [Knowledge-graph and structured-knowledge RAG](https://mlsanity.com/rag/knowledge-graph-and-structured-knowledge-rag/)

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models](https://mlsanity.com/paper/1802.08773/) (2018)
- [Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP](https://mlsanity.com/paper/2212.14024/) (2022)
- [InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions](https://mlsanity.com/paper/2304.05684/) (2023)
- [MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval](https://mlsanity.com/paper/2307.00589/) (2023)
- [Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modeling](https://mlsanity.com/paper/2304.02806/) (2023)
- [Knowledge Graph-Augmented Language Models for Knowledge-Grounded Dialogue Generation](https://mlsanity.com/paper/2305.18846/) (2023)
- [CycleGT: Unsupervised Graph-to-Text and Text-to-Graph Generation via Cycle Training](https://mlsanity.com/paper/2006.04702/) (2020)
- [Medical Exam Question Answering with Large-scale Reading Comprehension](https://mlsanity.com/paper/1802.10279/) (2018)
- [EASE: Entity-Aware Contrastive Learning of Sentence Embedding](https://mlsanity.com/paper/2205.04260/) (2022)

## Built on by (most cited first, 23 of 23)

- [A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models](https://mlsanity.com/paper/2501.13958/) (2025)
- [Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely](https://mlsanity.com/paper/2409.14924/) (2024)
- [UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities](https://mlsanity.com/paper/2504.20734/) (2025)
- [PIKE-RAG: sPecIalized KnowledgE and Rationale Augmented Generation](https://mlsanity.com/paper/2501.11551/) (2025)
- [Rethinking Chunk Size For Long-Document Retrieval: A Multi-Dataset Analysis](https://mlsanity.com/paper/2505.21700/) (2025)
- [FinGEAR: Financial Mapping-Guided Enhanced Answer Retrieval](https://mlsanity.com/paper/2509.12042/) (2025)
- [RAG Playground: A Framework for Systematic Evaluation of Retrieval Strategies and Prompt Engineering in RAG Systems](https://mlsanity.com/paper/2412.12322/) (2024)
- [A Survey of LLM × DATA](https://mlsanity.com/paper/2505.18458/) (2025)
- [SemRAG: Semantic Knowledge-Augmented RAG for Improved Question-Answering](https://mlsanity.com/paper/2507.21110/) (2025)
- [Dynamic Context Adaptation for Consistent Role-Playing Agents with Retrieval-Augmented Generations](https://mlsanity.com/paper/2508.02016/) (2025)
- [Hippocampus: An Efficient and Scalable Memory Module for Agentic AI](https://mlsanity.com/paper/2602.13594/) (2026)
- [Comparing Knowledge Injection Methods for LLMs in a Low-Resource Regime](https://mlsanity.com/paper/2508.06178/) (2025)
- [Are Finer Citations Always Better? Rethinking Granularity for Attributed Generation](https://mlsanity.com/paper/2604.01432/) (2026)
- [Advanced ingestion process powered by LLM parsing for RAG system](https://mlsanity.com/paper/2412.15262/) (2024)
- [Smooth Reading: Bridging the Gap of Recurrent LLM to Self-Attention LLM on Long-Context Tasks](https://mlsanity.com/paper/2507.19353/) (2025)
- [Failure is Feedback: History-Aware Backtracking for Agentic Traversal in Multimodal Graphs](https://mlsanity.com/paper/2602.03432/) (2026)
- [PersianRAG: A Retrieval-Augmented Generation System for Persian Language](https://mlsanity.com/paper/2411.02832/) (2024)
- [FreeChunker: A Cross-Granularity Chunking Framework](https://mlsanity.com/paper/2510.20356/) (2025)
- [Chunking Strategies for Multimodal AI Systems](https://mlsanity.com/paper/2512.00185/) (2025)
- [Breaking It Down: Domain-Aware Semantic Segmentation for Retrieval Augmented Generation](https://mlsanity.com/paper/2512.00367/) (2025)
- [TopoChunker: Topology-Aware Agentic Document Chunking Framework](https://mlsanity.com/paper/2603.18409/) (2026)
- [LLM-Based Intelligent Notification Composition: From Static Personalization to Context-Aware Persuasive Messaging](https://mlsanity.com/paper/2605.16264/) (2026)
- [Semantic Compression Trees: Multi-Resolution Knowledge Retrieval via Hierarchical Semantic Residuals](https://mlsanity.com/paper/2608.21610/) (2026)

## Abstract (arXiv)

Integrating information from various reference databases is a major challenge for Retrieval-Augmented Generation (RAG) systems because each knowledge source adopts a unique data structure and follows different conventions. Retrieving from multiple knowledge sources with one fixed strategy usually leads to under-exploitation of information. To mitigate this drawback, inspired by Mix-of-Expert, we introduce Mix-of-Granularity (MoG), a method that dynamically determines the optimal granularity of a knowledge source based on input queries using a router. The router is efficiently trained with a newly proposed loss function employing soft labels. We further extend MoG to MoG-Graph (MoGG), where reference documents are pre-processed as graphs, enabling the retrieval of distantly situated snippets. Experiments demonstrate that MoG and MoGG effectively predict optimal granularity levels, significantly enhancing the performance of the RAG system in downstream tasks. The code of both MoG and MoGG are released in https://github.com/ZGChung/Mix-of-Granularity.