# An Analysis of Hyper-Parameter Optimization Methods for Retrieval Augmented Generation

arXiv: https://arxiv.org/abs/2505.03452 · HTML: https://mlsanity.com/paper/2505.03452/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 16 of its 37 references that are in the map.
- First posted on arXiv 6 May 2025.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 16 of 16)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [Multilingual E5 Text Embeddings: A Technical Report](https://mlsanity.com/paper/2402.05672/) (2024)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization](https://mlsanity.com/paper/2109.09831/) (2021)
- [Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering](https://mlsanity.com/paper/2307.16877/) (2023)
- [Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance](https://mlsanity.com/paper/2304.11127/) (2023)
- [RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework](https://mlsanity.com/paper/2408.01262/) (2024)
- [Genie: Achieving Human Parity in Content-Grounded Datasets Generation](https://mlsanity.com/paper/2401.14367/) (2024)
- [AutoRAG: Automated Framework for optimization of Retrieval Augmented Generation Pipeline](https://mlsanity.com/paper/2410.20878/) (2024)
- [CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systems](https://mlsanity.com/paper/2404.02103/) (2024)
- [Faster, Cheaper, Better: Multi-Objective Hyperparameter Optimization for LLM and RAG Systems](https://mlsanity.com/paper/2502.18635/) (2025)

## Built on by (most cited first, 2 of 2)

- [CDS4RAG: Cyclic Dual-Sequential Hyperparameter Optimization for RAG](https://mlsanity.com/paper/2605.08333/) (2026)
- [Agentic AutoRAG: RAG Pipeline Optimization through Reasoning-Driven Agents](https://mlsanity.com/paper/2610.08452/) (2026)

## Abstract (arXiv)

Optimizing Retrieval-Augmented Generation (RAG) configurations for specific tasks is a complex and resource-intensive challenge. Motivated by this challenge, frameworks for RAG hyper-parameter optimization (HPO) have recently emerged, yet their effectiveness has not been rigorously benchmarked. To fill this gap, we present a comprehensive study involving five HPO algorithms over five datasets from diverse domains, including a newly curated real-world product documentation dataset. Our study explores the largest RAG HPO search space to date that includes full grid-search evaluations, and uses three evaluation metrics as optimization targets. Analysis of the results shows that RAG HPO can be done efficiently, either greedily or with random search, and that it significantly boosts RAG performance for all datasets. For greedy HPO approaches, we show that optimizing model selection first is preferable to the common practice of following the RAG pipeline order during optimization.