mlsanityData to 7 Oct 2026

RA-DIT: Retrieval-Augmented Dual Instruction Tuning

arXiv 2310.01352 ↗First posted 2 Oct 2023cs.CL cs.AIcited by 194

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Retrieval-augmented language models (RALMs) improve performance by accessing long-tail and up-to-date knowledge from external data stores, but are challenging to build. Existing approaches require either expensive retrieval-specific modifications to LM pre-training or use post-hoc integration of the data store that leads to suboptimal performance. We introduce Retrieval-Augmented Dual Instruction Tuning (RA-DIT), a lightweight fine-tuning methodology that provides a third option by retrofitting any LLM with retrieval capabilities. Our approach operates in two distinct fine-tuning steps: (1) one updates a pre-trained LM to better use retrieved information, while (2) the other updates the retriever to return more relevant results, as preferred by the LM. By fine-tuning over tasks that require both knowledge utilization and contextual awareness, we demonstrate that each stage yields significant performance improvements, and using both leads to additional gains. Our best model, RA-DIT 65B, achieves state-of-the-art performance across a range of knowledge-intensive zero- and few-shot learning benchmarks, significantly outperforming existing in-context RALM approaches by up to +8.9% in 0-shot setting and +1.4% in 5-shot setting on average.

Key facts. Cited by 194 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 62 of its 84 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

62 of 84 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023In-Context Retrieval-Augmented Language Models31 Jan
  2. 2023REPLUG: Retrieval-Augmented Black-Box Language Models30 Jan
  3. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories20 Dec
  4. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Models5 Aug
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 56

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  9. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  11. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  12. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  13. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  14. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  15. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  16. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  17. 2015Teaching Machines to Read and Comprehendcited by 1,039
  18. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  19. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  20. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  21. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  22. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  23. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  24. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  25. 2018CoQA: A Conversational Question Answering Challengecited by 674
  26. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  27. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  28. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  29. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  30. 2019ELI5: Long Form Question Answeringcited by 567
  31. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  32. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  33. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  34. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  35. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 283
  36. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  37. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  38. 2023Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?cited by 167

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 90; 2025: 65; 2026: 34.

194 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  3. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,000
  4. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  5. 2024The Faiss librarycited by 370
  6. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  7. 2024Generative Representational Instruction Tuningcited by 227
  8. 2025BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agentcited by 183
  9. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  10. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  11. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 122
  12. 2024ChatQA: Surpassing GPT-4 on Conversational QA and RAGcited by 90
38 more
  1. 2024Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Trainingcited by 88
  2. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  3. 2024Searching for Best Practices in Retrieval-Augmented Generationcited by 83
  4. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  5. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 76
  6. 2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
  7. 2024Instruction-tuned Language Models are Better Knowledge Learnerscited by 54
  8. 2024Scaling Retrieval-Based Language Models with a Trillion-Token Datastorecited by 54
  9. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmarkcited by 53
  10. 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Trainingcited by 50
  11. 2024LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answeringcited by 49
  12. 2025Chain-of-Retrieval Augmented Generationcited by 48
  13. 2024Evaluating Copyright Takedown Methods for Language Modelscited by 42
  14. 2024Modular RAG: Transforming RAG Systems into LEGO-like Reconfigurable Frameworkscited by 41
  15. 2025s3: You Don't Need That Much Data to Train a Search Agent via RLcited by 41
  16. 2024IM-RAG: Multi-Round Retrieval-Augmented Generation Through Learning Inner Monologuescited by 40
  17. 2025Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMscited by 40
  18. 2024M-RAG: Reinforcing Large Language Model Performance through Retrieval-Augmented Generation with Multiple Partitionscited by 38
  19. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 34
  20. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 32
  21. 2024JMLR: Joint Medical LLM and Retrieval Training for Enhancing Reasoning and Professional Question Answering Capabilitycited by 32
  22. 2025The Distracting Effect: Understanding Irrelevant Passages in RAGcited by 32
  23. 2024BERGEN: A Benchmarking Library for Retrieval-Augmented Generationcited by 31
  24. 2024REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answeringcited by 28
  25. 2024SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domainscited by 28
  26. 2025SurveyX: Academic Survey Automation via Large Language Modelscited by 27
  27. 2024Nearest Neighbor Speculative Decoding for LLM Generation and Attributioncited by 26
  28. 2024Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Passcited by 26
  29. 2025G-Refer: Graph Retrieval-Augmented Large Language Model for Explainable Recommendationcited by 26
  30. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processingcited by 25
  31. 2024Generating Is Believing: Membership Inference Attacks against Retrieval-Augmented Generationcited by 25
  32. 2024RAG-Instruct: Boosting LLMs with Diverse Retrieval-Augmented Instructionscited by 25
  33. 2024Learning to Plan for Retrieval-Augmented Large Language Models from Knowledge Graphscited by 24
  34. 2024RAP: Retrieval-Augmented Personalization for Multimodal Large Language Modelscited by 24
  35. 2024Synergistic Multi-Agent Framework with Trajectory Learning for Knowledge-Intensive Taskscited by 23
  36. 2024Data Extraction Attacks in Retrieval-Augmented Generation via Backdoorscited by 23
  37. 2025Retrieval Augmented Generation with Collaborative Filtering for Personalized Text Generationcited by 23
  38. 2024Self-MoE: Towards Compositional Large Language Models with Self-Specialized Expertscited by 22

50 most cited of 194 All 194 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining
  2. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Context
  3. 2024Accelerating Retrieval-Augmented Language Model Serving with Speculation
  4. 2025Do Retrieval-Augmented Language Models Adapt to Varying User Needs?
  5. 2026To Memorize or to Retrieve: Scaling the Interaction Between Pretraining and Retrieval
  6. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  7. 2023RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models
  8. 2025Instruction Tuning with and without Context: Behavioral Shifts and Downstream Impact