mlsanityData to 14 Oct 2026

RAG Playground: A Framework for Systematic Evaluation of Retrieval Strategies and Prompt Engineering in RAG Systems

arXiv 2412.12322 ↗First posted 16 Dec 2024cs.LG cs.AI cs.CLcited by 4

On the map: Retrieval-augmented generation → Agentic RAG, agents and memory

Abstract (arXiv)

We present RAG Playground, an open-source framework for systematic evaluation of Retrieval-Augmented Generation (RAG) systems. The framework implements and compares three retrieval approaches: naive vector search, reranking, and hybrid vector-keyword search, combined with ReAct agents using different prompting strategies. We introduce a comprehensive evaluation framework with novel metrics and provide empirical results comparing different language models (Llama 3.1 and Qwen 2.5) across various retrieval configurations. Our experiments demonstrate significant performance improvements through hybrid search methods and structured self-evaluation prompting, achieving up to 72.7% pass rate on our multi-metric evaluation framework. The results also highlight the importance of prompt engineering in RAG systems, with our custom-prompted agents showing consistent improvements in retrieval accuracy and response quality.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Agentic RAG, agents and memory, one of 15 directions in Retrieval-augmented generation. Builds on 19 of its 37 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

19 of 37 references.

References · 19

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  7. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  8. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 418
  9. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 287
  10. 2023Challenges and Applications of Large Language Modelscited by 254
  11. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 101
  12. 2020Language Models are Open Knowledge Graphscited by 95
7 more
  1. 2024RAGEval: Scenario Specific RAG Evaluation Dataset Generation Frameworkcited by 39
  2. 2024The Chronicles of RAG: The Retriever, the Chunk and the Generatorcited by 34
  3. 2020Leveraging Semantic and Lexical Matching to Improve the Recall of Document Retrieval Systems: A Hybrid Approachcited by 32
  4. 2022Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methodscited by 27
  5. 2024Mix-of-Granularity: Optimize the Chunking Granularity for Retrieval-Augmented Generationcited by 23
  6. 2024Introducing a new hyper-parameter for RAG: Context Window Utilizationcited by 8
  7. 2016Neural Information Retrieval: A Literature Reviewcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Surveycited by 20
  2. 2025Human-Centered Evaluation of RAG outputs: a framework and questionnaire for human-AI collaborationcited by 1
  3. 2026Still Fresh? Evaluating Temporal Drift in Retrieval Benchmarkscited by 1
  4. 2026Beyond Aggregate Scores: Behavioral Correctness Assumptions for Assessing Reference-Based Automatic Evaluation Methodscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Power of Noise: Redefining Retrieval for RAG Systems
  2. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoning
  3. 2025Enhancing Retrieval-Augmented Generation: A Study of Best Practices
  4. 2026Is Agentic RAG worth it? An experimental comparison of RAG approaches
  5. 2025Keyword search is all you need: Achieving RAG-Level Performance without vector databases using agentic tool use
  6. 2025RAGVUE: A Diagnostic View for Explainable and Automated Evaluation of Retrieval-Augmented Generation
  7. 2024Toward Optimal Search and Retrieval for RAG
  8. 2026A-RAG: Scaling Agentic Retrieval-Augmented Generation via Hierarchical Retrieval Interfaces