mlsanityData to 7 Oct 2026

Evaluating Hybrid Retrieval Augmented Generation using Dynamic Test Sets: LiveRAG Challenge

arXiv 2506.22644 ↗First posted 27 Jun 2025cs.CL cs.IRcited by 3

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

We present our submission to the LiveRAG Challenge 2025, which evaluates retrieval-augmented generation (RAG) systems on dynamic test sets using the FineWeb-10BT corpus. Our final hybrid approach combines sparse (BM25) and dense (E5) retrieval methods and then aims to generate relevant and faithful answers with Falcon3-10B-Instruct. Through systematic evaluation on 200 synthetic questions generated with DataMorgana across 64 unique question-user combinations, we demonstrate that neural re-ranking with RankLLaMA improves MAP from 0.523 to 0.797 (52% relative improvement) but introduces prohibitive computational costs (84s vs 1.74s per question). While DSPy-optimized prompting strategies achieved higher semantic similarity (0.771 vs 0.668), their 0% refusal rates raised concerns about over-confidence and generalizability. Our submitted hybrid system without re-ranking achieved 4th place in faithfulness and 11th place in correctness among 25 teams. Analysis across question categories reveals that vocabulary alignment between questions and documents was the strongest predictor of performance on our development set, with document-similar phrasing improving cosine similarity from 0.562 to 0.762.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 6 of its 14 references that are in the map. First posted on arXiv 27 Jun 2025.

Builds on

6 of 14 references.

References · 6

  1. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 271
  2. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 270
  3. 2019Document Expansion by Query Predictioncited by 225
  4. 2023RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!cited by 151
  5. 2024Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Togethercited by 38
  6. 2025Generating Diverse Q&A Benchmarks for RAG Evaluation with DataMorganacited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025SINDI: An Efficient Index for Sparse Vector Approximate Maximum Inner Product Searchcited by 2
  2. 2026RATE: Reviewer Profiling and Annotation-free Training for Expertise Ranking in Peer Review Systemscited by 1
  3. 2026Approximate Nearest Neighbor Search for Modern AI: A Projection-Augmented Graph Approachcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  2. 2026A Systems-Level Analysis of Sensitivity, Robustness, and Stability in Retrieval-Augmented Generation
  3. 2025Optimizing Retrieval-Augmented Generation: Analysis of Hyperparameter Impact on Performance and Efficiency
  4. 2026SciRet: A Compute-Aware Empirical Study of Retrieval and Reranking for Scientific RAG
  5. 2025RAGSmith: A Framework for Finding the Optimal Composition of Retrieval-Augmented Generation Methods Across Datasets
  6. 2026A Reproducibility Study of Metacognitive Retrieval-Augmented Generation
  7. 2025PreQRAG -- Classify and Rewrite for Enhanced RAG
  8. 2026Predict the Retrieval! Test time adaptation for Retrieval Augmented Generation