mlsanityData to 14 Oct 2026

RAFT: Adapting Language Model to Domain Specific RAG

arXiv 2403.10131 ↗First posted 15 Mar 2024cs.CL cs.AIcited by 262

Not on the map. Why?

Cited by 149 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pretraining Large Language Models (LLMs) on large corpora of textual data is now a standard paradigm. When using these LLMs for many downstream applications, it is common to additionally bake in new knowledge (e.g., time-critical news, or private domain knowledge) into the pretrained model either through RAG-based-prompting, or fine-tuning. However, the optimal methodology for the model to gain such new knowledge remains an open question. In this paper, we present Retrieval Augmented FineTuning (RAFT), a training recipe that improves the model's ability to answer questions in a "open-book" in-domain settings. In RAFT, given a question, and a set of retrieved documents, we train the model to ignore those documents that don't help in answering the question, which we call, distractor documents. RAFT accomplishes this by citing verbatim the right sequence from the relevant document that would help answer the question. This coupled with RAFT's chain-of-thought-style response helps improve the model's ability to reason. In domain-specific RAG, RAFT consistently improves the model's performance across PubMed, HotpotQA, and Gorilla datasets, presenting a post-training recipe to improve pre-trained LLMs to in-domain RAG. RAFT's code and demo are open-sourced at github.com/ShishirPatil/gorilla.

Key facts. Cited by 262 papers in the map (data through 14 Oct 2026). Builds on 48 of its 57 references that are in the map. First posted on arXiv 15 Mar 2024.

Builds on

48 of 57 references.

References · 48

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
36 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  3. 2020Extracting Training Data from Large Language Modelscited by 2,189
  4. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  6. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  8. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  9. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  10. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  11. 2023PaLM 2 Technical Reportcited by 1,162
  12. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  13. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  14. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  15. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  16. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  17. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  18. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  19. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  20. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  21. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
  22. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  23. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  24. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  25. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  26. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
  27. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 195
  28. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 148
  29. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  30. 2017PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstractscited by 124
  31. 2023Retrieval meets Long Context Large Language Modelscited by 115
  32. 2023System 2 Attention (is something you might need too)cited by 89
  33. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 79
  34. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 62
  35. 2023InstructRetro: Instruction Tuning post Retrieval-Augmented Pretrainingcited by 53
  36. 2021Memorisation versus Generalisation in Pre-trained Language Modelscited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 102; 2025: 102; 2026: 58.

262 papers.

Sort:

By area:

  1. 2024Capabilities of Gemini Models in Medicinecited by 237
  2. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 202
  3. 2025Dynamic Cheatsheet: Test-Time Learning with Adaptive Memorycited by 163
  4. 2024Certifiably Robust RAG against Retrieval Corruptioncited by 151
  5. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  6. 2024GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoningcited by 128
  7. 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 124
  8. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 123
  9. 2024InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationalescited by 101
  10. 2024Searching for Best Practices in Retrieval-Augmented Generationcited by 83
  11. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directionscited by 70
  12. 2024Evaluation of Text-to-Video Generation Models: A Dynamics Perspectivecited by 67
38 more
  1. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 66
  2. 2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
  3. 2025ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templatescited by 63
  4. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmarkcited by 55
  5. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  6. 2024LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answeringcited by 49
  7. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  8. 2025Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMscited by 42
  9. 2024Rationale-Guided Retrieval Augmented Generation for Medical Question Answeringcited by 41
  10. 2024BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrieverscited by 38
  11. 2024M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Frameworkcited by 36
  12. 2024How Much Can RAG Help the Reasoning of LLM?cited by 31
  13. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directionscited by 31
  14. 2025RAG-RL: Advancing Retrieval-Augmented Generation via RL and Curriculum Learningcited by 30
  15. 2024SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domainscited by 28
  16. 2024Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generationcited by 27
  17. 2025G-Refer: Graph Retrieval-Augmented Large Language Model for Explainable Recommendationcited by 27
  18. 2025MA-RAG: Multi-Agent Retrieval-Augmented Generation via Collaborative Chain-of-Thought Reasoningcited by 26
  19. 2024Open-RAG: Enhanced Retrieval-Augmented Reasoning with Open-Source Large Language Modelscited by 25
  20. 2024RAG-Instruct: Boosting LLMs with Diverse Retrieval-Augmented Instructionscited by 25
  21. 2024Efficient Knowledge Injection in LLMs via Self-Distillationcited by 24
  22. 2024Data Extraction Attacks in Retrieval-Augmented Generation via Backdoorscited by 23
  23. 2024RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignmentcited by 23
  24. 2024OG-RAG: Ontology-Grounded Retrieval-Augmented Generation For Large Language Modelscited by 23
  25. 2024Can Large Language Models abstract Medical Coded Language?cited by 20
  26. 2025Collab-RAG: Boosting Retrieval-Augmented Generation for Complex Question Answering via White-Box and Black-Box LLM Collaborationcited by 20
  27. 2024ATM: Adversarial Tuning Multi-agent System Makes a Robust Retrieval-Augmented Generatorcited by 19
  28. 2024Entropy-Based Decoding for Retrieval-Augmented Large Language Modelscited by 19
  29. 2024RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialoguescited by 19
  30. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  31. 2025Fine-tuning a Large Language Model for Automating Computational Fluid Dynamics Simulationscited by 18
  32. 2026A Survey on LLM-based Conversational User Simulationcited by 18
  33. 2024Question-Based Retrieval using Atomic Units for Enterprise RAGcited by 17
  34. 2024Self-Tuning: Instructing LLMs to Effectively Acquire New Knowledge through Self-Teachingcited by 17
  35. 2024SEC-QA: A Systematic Evaluation Corpus for Financial QAcited by 17
  36. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 17
  37. 2024LLoCO: Learning Long Contexts Offlinecited by 16
  38. 2024DomainRAG: A Chinese Benchmark for Evaluating Domain-specific Retrieval-Augmented Generationcited by 16

50 most cited of 262 All 262 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Domain-Specific Retrieval-Augmented Generation Using Vector Stores, Knowledge Graphs, and Tensor Factorization
  2. 2024Retrieval-Augmented Generation for Domain-Specific Question Answering: A Case Study on Pittsburgh and CMU
  3. 2025Rethinking Data: Towards Better Performing Domain-Specific Small Language Models
  4. 2024Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge
  5. 2022Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering
  6. 2024W-RAG: Weakly Supervised Dense Retrieval in RAG for Open-domain Question Answering
  7. 2024VERA: Validation and Enhancement for Retrieval Augmented systems
  8. 2025LMAR: Language Model Augmented Retriever for Domain-specific Knowledge Indexing