mlsanityData to 14 Oct 2026

RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models

arXiv 2308.10633 ↗First posted 21 Aug 2023cs.CL cs.AIcited by 4

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Retrieval-augmented large language models (R-LLMs) combine pre-trained large language models (LLMs) with information retrieval systems to improve the accuracy of factual question-answering. However, current libraries for building R-LLMs provide high-level abstractions without sufficient transparency for evaluating and optimizing prompts within specific inference processes such as retrieval and generation. To address this gap, we present RaLLe, an open-source framework designed to facilitate the development, evaluation, and optimization of R-LLMs for knowledge-intensive tasks. With RaLLe, developers can easily develop and evaluate R-LLMs, improving hand-crafted prompts, assessing individual inference processes, and objectively measuring overall system performance quantitatively. By leveraging these features, developers can enhance the performance and accuracy of their R-LLMs in knowledge-intensive generation tasks. We open-source our code at https://github.com/yhoshi3/RaLLe.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 31 of its 47 references that are in the map. First posted on arXiv 21 Aug 2023.

Builds on

31 of 47 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023REPLUG: Retrieval-Augmented Black-Box Language Models30 Jan
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  10. 2020Learning to summarize from human feedbackcited by 2,588
  11. 2017Billion-scale similarity search with GPUscited by 2,497
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
16 more
  1. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  2. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  3. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  4. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  5. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
  6. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  7. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  8. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
  9. 2023Augmented Language Models: a Surveycited by 403
  10. 2019Gradio: Hassle-Free Sharing and Testing of ML Models in the Wildcited by 178
  11. 2023A Categorical Archive of ChatGPT Failurescited by 168
  12. 2022StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Modelscited by 126
  13. 2020Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queriescited by 107
  14. 2020Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrievalcited by 44
  15. 2022What Are You Token About? Dense Retrieval as Distributions Over the Vocabularycited by 28
  16. 2023SimplyRetrieve: A Private and Lightweight Retrieval-Centric Generative AI Toolcited by 6

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3.

4 papers.

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  2. 2024ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidencecited by 113
  3. 2024DMQR-RAG: Diverse Multi-Query Rewriting for RAGcited by 27
  4. 2024XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generationcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing
  2. 2024R-Eval: A Unified Toolkit for Evaluating Domain Knowledge of Retrieval Augmented Large Language Models
  3. 2024FlashBack: Efficient Retrieval-Augmented Language Modeling for Fast Inference
  4. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Context
  5. 2023RETA-LLM: A Retrieval-Augmented Large Language Model Toolkit
  6. 2024Reliable, Adaptable, and Attributable Language Models with Retrieval
  7. 2023In-Context Retrieval-Augmented Language Models
  8. 2024RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialogues