mlsanityData to 14 Oct 2026

Constructing and Evaluating Declarative RAG Pipelines in PyTerrier

arXiv 2506.10802 ↗First posted 12 Jun 2025cs.IRcited by 5

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Search engines often follow a pipeline architecture, where complex but effective reranking components are used to refine the results of an initial retrieval. Retrieval augmented generation (RAG) is an exciting application of the pipeline architecture, where the final component generates a coherent answer for the users from the retrieved documents. In this demo paper, we describe how such RAG pipelines can be formulated in the declarative PyTerrier architecture, and the advantages of doing so. Our PyTerrier-RAG extension for PyTerrier provides easy access to standard RAG datasets and evaluation measures, state-of-the-art LLM readers, and using PyTerrier's unique operator notation, easy-to-build pipelines. We demonstrate the succinctness of indexing and RAG pipelines on standard datasets (including Natural Questions) and how to build on the larger PyTerrier ecosystem with state-of-the-art sparse, learned-sparse, and dense retrievers, and other neural rankers.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 36 of its 51 references that are in the map. First posted on arXiv 12 Jun 2025.

Builds on

36 of 51 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Is Relevance Propagated from Retriever to Generator in RAG?20 Feb
  2. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research22 May
  3. 2024The Power of Noise: Redefining Retrieval for RAG Systems26 Jan

Other references · 33

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  8. 2017Billion-scale similarity search with GPUscited by 2,497
  9. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  10. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  11. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  12. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
21 more
  1. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  2. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  3. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  4. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  5. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
  6. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  7. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
  8. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  9. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  10. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  11. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 346
  12. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 286
  13. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 267
  14. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 195
  15. 2021The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Modelscited by 81
  16. 2024UniMS-RAG: A Unified Multi-source Retrieval-Augmented Generation for Personalized Dialogue Systemscited by 38
  17. 2021Pseudo-Relevance Feedback for Multiple Representation Dense Retrievalcited by 32
  18. 2020Declarative Experimentation in Information Retrieval using PyTerriercited by 30
  19. 2021Pseudo Relevance Feedback with Deep Language Models and Dense Retrievers: Successes and Pitfallscited by 26
  20. 2025On Precomputation and Caching in Information Retrieval Experiments with Pipeline Architecturescited by 1
  21. 2025Artifact Sharing for Information Retrieval Researchcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domaincited by 6
  2. 2025Am I on the Right Track? What Can Predicted Query Performance Tell Us about the Search Behaviour of Agentic RAGcited by 5
  3. 2026RoutIR: Fast Serving of Retrieval Pipelines for Retrieval-Augmented Generationcited by 1
  4. 2026Predicting Partial Answer Quality and Utility in Agentic Retrieval-Augmented Generationcited by 1
  5. 2026Guaranteeing Knowledge Integration with Joint Decoding for Retrieval-Augmented Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Toward Optimal Search and Retrieval for RAG
  2. 2026LURE-RAG: Lightweight Utility-driven Reranking for Efficient RAG
  3. 2026Sifei at SemEval-2026 Task 8: Hybrid Retrieval and Query Rewriting for Multi-Turn RAG
  4. 2024CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity
  5. 2025RAGSmith: A Framework for Finding the Optimal Composition of Retrieval-Augmented Generation Methods Across Datasets
  6. 2025PreQRAG -- Classify and Rewrite for Enhanced RAG
  7. 2025LevelRAG: Enhancing Retrieval-Augmented Generation with Multi-hop Logic Planning over Rewriting Augmented Searchers
  8. 2026RAISE: RAG Design as an Architecture Search Problem