# CDS4RAG: Cyclic Dual-Sequential Hyperparameter Optimization for RAG

arXiv: https://arxiv.org/abs/2605.08333 · HTML: https://mlsanity.com/paper/2605.08333/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 37 of its 45 references that are in the map.
- First posted on arXiv 8 May 2026.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 30 of 37)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning](https://mlsanity.com/paper/2507.19457/) (2025)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity](https://mlsanity.com/paper/2403.14403/) (2024)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [LightRAG: Simple and Fast Retrieval-Augmented Generation](https://mlsanity.com/paper/2410.05779/) (2024)
- [Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs](https://mlsanity.com/paper/2406.11695/) (2024)
- [RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models](https://mlsanity.com/paper/2401.00396/) (2023)
- [Evaluation of Retrieval-Augmented Generation: A Survey](https://mlsanity.com/paper/2405.07437/) (2024)
- [A Review on Bilevel Optimization: From Classical to Evolutionary Approaches and Applications](https://mlsanity.com/paper/1705.06270/) (2017)
- [Evaluating Retrieval Quality in Retrieval-Augmented Generation](https://mlsanity.com/paper/2404.13781/) (2024)
- [HEBO Pushing The Limits of Sample-Efficient Hyperparameter Optimisation](https://mlsanity.com/paper/2012.03826/) (2020)
- [AutoML to Date and Beyond: Challenges and Opportunities](https://mlsanity.com/paper/2010.10777/) (2020)
- [Multi-Objectivizing Software Configuration Tuning (for a single performance concern)](https://mlsanity.com/paper/2106.01331/) (2021)
- [MMO: Meta Multi-Objectivization for Software Configuration Tuning](https://mlsanity.com/paper/2112.07303/) (2021)
- [Predicting Configuration Performance in Multiple Environments with Sequential Meta-learning](https://mlsanity.com/paper/2402.03183/) (2024)
- [Rag and Roll: An End-to-End Evaluation of Indirect Prompt Manipulations in LLM-based Application Frameworks](https://mlsanity.com/paper/2408.05025/) (2024)
- [METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation](https://mlsanity.com/paper/2412.10543/) (2024)
- [Lifelong Dynamic Optimization for Self-Adaptive Systems: Fact or Fiction?](https://mlsanity.com/paper/2201.07096/) (2022)
- [Predicting Software Performance with Divide-and-Learn](https://mlsanity.com/paper/2306.06651/) (2023)
- [Adapting Multi-objectivized Software Configuration Tuning](https://mlsanity.com/paper/2404.04744/) (2024)
- [AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for Retrieval-Augmented Generation](https://mlsanity.com/paper/2406.19251/) (2024)
- [Distilled Lifelong Self-Adaptation for Configurable Systems](https://mlsanity.com/paper/2501.00840/) (2025)
- [Faster, Cheaper, Better: Multi-Objective Hyperparameter Optimization for LLM and RAG Systems](https://mlsanity.com/paper/2502.18635/) (2025)
- [PromiseTune: Unveiling Causally Promising and Explainable Configuration Tuning](https://mlsanity.com/paper/2507.05995/) (2025)
- [Accuracy Can Lie: On the Impact of Surrogate Model in Configuration Tuning](https://mlsanity.com/paper/2501.01876/) (2025)

## Built on by (most cited first, 1 of 1)

- [Configuring RAG Systems: What Breaks, Where Originates, and How to Fix](https://mlsanity.com/paper/2610.18567/) (2026)

## Abstract (arXiv)

Retrieval-Augmented Generation (RAG) is sensitive to the vast hyperparameters of the retriever and generator, yet optimizing them using given queries is a challenging task due to the complex interactions and expensive evaluation costs. Existing algorithms are ineffective and slow in convergence, since they often treat RAG as a monolithic black box or only optimize partial hyperparameters. In this paper, we propose CDS4RAG, a framework that optimizes the full RAG hyperparameters using given queries via a new cyclic dual-sequential formulation. CDS4RAG is special in the sense that it distinguishes the hyperparameters of the retriever and generator, cyclically optimizing them in turn. Such a paradigm allows us to design fine-grained within-cycle budget provision and expedite the optimization via cross-cycle seeding when optimizing the generator. CDS4RAG is also an algorithm-agnostic framework that can be paired with diverse general algorithms. Through experiments on four common benchmarks and two backbone LLMs, we reveal that CDS4RAG considerably boosts the vanilla algorithms in 21/24 cases while significantly outperforming state-of-the-art algorithms in all cases with up to 1.54x improvements of generation quality and better speedup.