# Evaluating Hybrid Retrieval Augmented Generation using Dynamic Test Sets: LiveRAG Challenge

arXiv: https://arxiv.org/abs/2506.22644 · HTML: https://mlsanity.com/paper/2506.22644/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation.
- Builds on 6 of its 14 references that are in the map.
- First posted on arXiv 27 Jun 2025.

## Placed in

- Retrieval-augmented generation / [RAG robustness and adaptive retrieval](https://mlsanity.com/rag/rag-robustness-and-adaptive-retrieval/)

## Builds on (most cited first, 6 of 6)

- [Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs](https://mlsanity.com/paper/2406.11695/) (2024)
- [Fine-Tuning LLaMA for Multi-Stage Text Retrieval](https://mlsanity.com/paper/2310.08319/) (2023)
- [Document Expansion by Query Prediction](https://mlsanity.com/paper/1904.08375/) (2019)
- [RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!](https://mlsanity.com/paper/2312.02724/) (2023)
- [Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together](https://mlsanity.com/paper/2407.10930/) (2024)
- [Generating Diverse Q&A Benchmarks for RAG Evaluation with DataMorgana](https://mlsanity.com/paper/2501.12789/) (2025)

## Built on by (most cited first, 3 of 3)

- [SINDI: An Efficient Index for Sparse Vector Approximate Maximum Inner Product Search](https://mlsanity.com/paper/2509.08395/) (2025)
- [RATE: Reviewer Profiling and Annotation-free Training for Expertise Ranking in Peer Review Systems](https://mlsanity.com/paper/2601.19637/) (2026)
- [Approximate Nearest Neighbor Search for Modern AI: A Projection-Augmented Graph Approach](https://mlsanity.com/paper/2603.06660/) (2026)

## Abstract (arXiv)

We present our submission to the LiveRAG Challenge 2025, which evaluates retrieval-augmented generation (RAG) systems on dynamic test sets using the FineWeb-10BT corpus. Our final hybrid approach combines sparse (BM25) and dense (E5) retrieval methods and then aims to generate relevant and faithful answers with Falcon3-10B-Instruct. Through systematic evaluation on 200 synthetic questions generated with DataMorgana across 64 unique question-user combinations, we demonstrate that neural re-ranking with RankLLaMA improves MAP from 0.523 to 0.797 (52% relative improvement) but introduces prohibitive computational costs (84s vs 1.74s per question). While DSPy-optimized prompting strategies achieved higher semantic similarity (0.771 vs 0.668), their 0% refusal rates raised concerns about over-confidence and generalizability. Our submitted hybrid system without re-ranking achieved 4th place in faithfulness and 11th place in correctness among 25 teams. Analysis across question categories reveals that vocabulary alignment between questions and documents was the strongest predictor of performance on our development set, with document-similar phrasing improving cosine similarity from 0.562 to 0.762.