mlsanityData to 14 Oct 2026

An Analysis of Hyper-Parameter Optimization Methods for Retrieval Augmented Generation

arXiv 2505.03452 ↗First posted 6 May 2025cs.CL cs.AI cs.LGcited by 2

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Optimizing Retrieval-Augmented Generation (RAG) configurations for specific tasks is a complex and resource-intensive challenge. Motivated by this challenge, frameworks for RAG hyper-parameter optimization (HPO) have recently emerged, yet their effectiveness has not been rigorously benchmarked. To fill this gap, we present a comprehensive study involving five HPO algorithms over five datasets from diverse domains, including a newly curated real-world product documentation dataset. Our study explores the largest RAG HPO search space to date that includes full grid-search evaluations, and uses three evaluation metrics as optimization targets. Analysis of the results shows that RAG HPO can be done efficiently, either greedily or with random search, and that it significantly boosts RAG performance for all datasets. For greedy HPO approaches, we show that optimizing model selection first is preferable to the common practice of following the RAG pipeline order during optimization.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 16 of its 37 references that are in the map. First posted on arXiv 6 May 2025.

Builds on

16 of 37 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Faster, Cheaper, Better: Multi-Objective Hyperparameter Optimization for LLM and RAG Systems25 Feb
  2. 2024AutoRAG: Automated Framework for optimization of Retrieval Augmented Generation Pipeline28 Oct
  3. 2024RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework2 Aug
  4. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec

Other references · 12

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  3. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
  4. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 470
  5. 2018Tune: A Research Platform for Distributed Model Selection and Trainingcited by 395
  6. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 372
  7. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  8. 2021SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimizationcited by 230
  9. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 151
  10. 2023Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performancecited by 147
  11. 2024Genie: Achieving Human Parity in Content-Grounded Datasets Generationcited by 31
  12. 2024CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systemscited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026CDS4RAG: Cyclic Dual-Sequential Hyperparameter Optimization for RAGcited by 1
  2. 2026Agentic AutoRAG: RAG Pipeline Optimization through Reasoning-Driven Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RAISE: RAG Design as an Architecture Search Problem
  2. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving
  3. 2026RAG-Stack: Co-Optimizing RAG Serving Performance and Quality
  4. 2024Towards Optimizing a Retrieval Augmented Generation using Large Language Model on Academic Data
  5. 2025RAG-Stack: Co-Optimizing RAG Quality and Performance From the Vector Database Perspective
  6. 2025RAGSmith: A Framework for Finding the Optimal Composition of Retrieval-Augmented Generation Methods Across Datasets
  7. 2025Optimizing Retrieval-Augmented Generation: Analysis of Hyperparameter Impact on Performance and Efficiency
  8. 2024Toward Optimal Search and Retrieval for RAG