Exploring Information Retrieval Landscapes: An Investigation of a Novel Evaluation Techniques and Comparative Document Splitting Methods
On the map: Retrieval-augmented generation → RAG pipeline design and evaluation
Abstract (arXiv)
The performance of Retrieval-Augmented Generation (RAG) systems in information retrieval is significantly influenced by the characteristics of the documents being processed. In this study, the structured nature of textbooks, the conciseness of articles, and the narrative complexity of novels are shown to require distinct retrieval strategies. A comparative evaluation of multiple document-splitting methods reveals that the Recursive Character Splitter outperforms the Token-based Splitter in preserving contextual integrity. A novel evaluation technique is introduced, utilizing an open-source model to generate a comprehensive dataset of question-and-answer pairs, simulating realistic retrieval scenarios to enhance testing efficiency and metric reliability. The evaluation employs weighted scoring metrics, including SequenceMatcher, BLEU, METEOR, and BERT Score, to assess the system's accuracy and relevance. This approach establishes a refined standard for evaluating the precision of RAG systems, with future research focusing on optimizing chunk and overlap sizes to improve retrieval accuracy and efficiency.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 6 of its 16 references that are in the map. First posted on arXiv 13 Sep 2024.
Builds on
6 of 16 references.
References · 6
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2020Longformer: The Long-Document Transformercited by 3,362
- 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
- 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
- 2026Evaluating Chunking Strategies for Retrieval-Augmented Generation on Academic Texts
- 2025Enhancing Retrieval-Augmented Generation: A Study of Best Practices
- 2024Evaluation of Retrieval-Augmented Generation: A Survey
- 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions
- 2024Evaluating Retrieval Quality in Retrieval-Augmented Generation
- 2024Is Semantic Chunking Worth the Computational Cost?
- 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generation