# ReBOL: Retrieval via Bayesian Optimization with Batched LLM Relevance Observations and Query Reformulation

arXiv: https://arxiv.org/abs/2603.20513 · HTML: https://mlsanity.com/paper/2603.20513/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 20 Mar 2026.

## Builds on (most cited first, 13 of 13)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [Improving Passage Retrieval with Zero-Shot Question Generation](https://mlsanity.com/paper/2204.07496/) (2022)
- [Condenser: a Pre-training Architecture for Dense Retrieval](https://mlsanity.com/paper/2104.08253/) (2021)
- [RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!](https://mlsanity.com/paper/2312.02724/) (2023)
- [UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor](https://mlsanity.com/paper/2406.06519/) (2024)
- [GenQREnsemble: Zero-Shot LLM Ensemble Prompting for Generative Query Reformulation](https://mlsanity.com/paper/2404.03746/) (2024)
- [A Surprisingly Simple yet Effective Multi-Query Rewriting Method for Conversational Passage Retrieval](https://mlsanity.com/paper/2406.18960/) (2024)
- [Systematic Evaluation of Neural Retrieval Models on the Touch'e 2020 Argument Retrieval Subset of BEIR](https://mlsanity.com/paper/2407.07790/) (2024)
- [Batched Self-Consistency Improves LLM Relevance Assessment and Ranking](https://mlsanity.com/paper/2505.12570/) (2025)
- [A Simple but Effective Elaborative Query Reformulation Approach for Natural Language Recommendation](https://mlsanity.com/paper/2510.02656/) (2025)
- [Multi-Aspect Reviewed-Item Retrieval via LLM Query Decomposition and Aspect Fusion](https://mlsanity.com/paper/2408.00878/) (2024)

## Abstract (arXiv)

LLM-reranking is limited by the top-k documents retrieved by vector similarity, which neither enables contextual query-document token interactions nor captures multimodal relevance distributions. While LLM query reformulation attempts to improve recall by generating improved or additional queries, it is still followed by vector similarity retrieval. We thus propose to address these top-k retrieval stage failures by introducing ReBOL, which 1) uses LLM query reformulations to initialize a multimodal Bayesian Optimization (BO) posterior over document relevance, and 2) iteratively acquires document batches for LLM query-document relevance scoring followed by posterior updates to optimize relevance. After exploring query reformulation and document batch diversification techniques, we evaluate ReBOL against LLM reranker baselines on five BEIR datasets and using two LLMs (Gemini-2.5-Flash-Lite, GPT-5.2). ReBOL consistently achieves higher recall and competitive rankings, for example compared to the best LLM reranker on the Robust04 dataset with 46.5% vs. 35.0% recall@100 and 63.6% vs. 61.2% NDCG@10. We also show that ReBOL can achieve comparable latency to LLM rerankers.