mlsanityData to 14 Oct 2026

SmartRAG: Jointly Learn RAG-Related Tasks From the Environment Feedback

arXiv 2410.18141 ↗First posted 22 Oct 2024cs.IR cs.AI cs.CLcited by 19

On the map: Retrieval-augmented generation → Agentic RAG, agents and memory

Abstract (arXiv)

RAG systems consist of multiple modules to work together. However, these modules are usually separately trained. We argue that a system like RAG that incorporates multiple modules should be jointly optimized to achieve optimal performance. To demonstrate this, we design a specific pipeline called SmartRAG that includes a policy network and a retriever. The policy network can serve as 1) a decision maker that decides when to retrieve, 2) a query rewriter to generate a query most suited to the retriever, and 3) an answer generator that produces the final response with/without the observations. We then propose to jointly optimize the whole system using a reinforcement learning algorithm, with the reward designed to encourage the system to achieve the best performance with minimal retrieval cost. When jointly optimized, all the modules can be aware of how other modules are working and thus find the best way to work together as a complete system. Empirical results demonstrate that the jointly optimized SmartRAG can achieve better performance than separately optimized counterparts.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Agentic RAG, agents and memory, one of 15 directions in Retrieval-augmented generation. Builds on 46 of its 48 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

46 of 48 references.

References · 46

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  8. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  12. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
34 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  3. 2020Learning to summarize from human feedbackcited by 2,588
  4. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  7. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  8. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  9. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  10. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  11. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  12. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  13. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
  14. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  15. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  16. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  17. 2023Active Retrieval Augmented Generationcited by 516
  18. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 451
  19. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  20. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  21. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 318
  22. 2023AI Alignment: A Comprehensive Surveycited by 263
  23. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  24. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 213
  25. 2023OpenAGI: When LLM Meets Domain Expertscited by 197
  26. 2022Re2G: Retrieve, Rerank, Generatecited by 102
  27. 2023Making Language Models Better Tool Learners with Execution Feedbackcited by 70
  28. 2023Self-Knowledge Guided Retrieval Augmentation for Large Language Modelscited by 69
  29. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57
  30. 2024RA-ISF: Learning to Answer and Understand from Retrieval Augmentation via Iterative Self-Feedbackcited by 46
  31. 2024Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMscited by 34
  32. 2022Learning Dexterous Manipulation from Exemplar Object Trajectories and Pre-Graspscited by 29
  33. 2023Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Makingcited by 25
  34. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processingcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 10.

19 papers.

Sort:

By area:

  1. 2025Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learningcited by 39
  2. 2025Synergizing RAG and Reasoning: A Systematic Reviewcited by 27
  3. 2025DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generationcited by 15
  4. 2025The Silent Saboteur: Imperceptible Adversarial Attacks against Black-Box Retrieval-Augmented Generation Systemscited by 10
  5. 2026How Many Tools Should an LLM Agent See? A Chance-Corrected Answercited by 7
  6. 2025Chain-of-Thought Poisoning Attacks against R1-based Retrieval-Augmented Generation Systemscited by 6
  7. 2025Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domaincited by 6
  8. 2026ElasticMem: Latent Memory as a Learnable Resource for LLM Agentscited by 4
  9. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  10. 2026AdversarialCoT: Single-Document Retrieval Poisoning for LLM Reasoningcited by 3
  11. 2025CardRewriter: Leveraging Knowledge Cards for Long-Tail Query Rewriting on Short-Video Platformscited by 2
  12. 2026LURE-RAG: Lightweight Utility-driven Reranking for Efficient RAGcited by 2
7 more
  1. 2026When Iterative RAG Beats Ideal Evidence: A Diagnostic Study in Scientific Multi-hop Question Answeringcited by 1
  2. 2026MG2-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generationcited by 1
  3. 2025Augmenting LLM Reasoning with Dynamic Notes Writing for Complex QAcited by 0
  4. 2025Learning What Helps: Task-Aligned Context Selection for Vision Taskscited by 0
  5. 2026Did You Check the Right Pocket? Cost-Sensitive Store Routing for Memory-Augmented Agentscited by 0
  6. 2026Beyond Relevance: Utility-Centric Retrieval in the LLM Eracited by 0
  7. 2026BRIDGE: Bilevel Retrieval-Credit-Aware Agentic Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimizing Retrieval for RAG via Reinforcement Learning
  2. 2025R3-RAG: Learning Step-by-Step Reasoning and Retrieval for LLMs via Reinforcement Learning
  3. 2025HiPRAG: Hierarchical Process Rewards for Efficient Agentic Retrieval Augmented Generation
  4. 2026Reinforcement Fine-Tuning for History-Aware Dense Retriever in RAG
  5. 2025Advancing Multi-Agent RAG Systems with Minimalist Reinforcement Learning
  6. 2025DecEx-RAG: Boosting Agentic Retrieval-Augmented Generation with Decision and Execution Optimization via Process Supervision
  7. 2025Curriculum Guided Reinforcement Learning for Efficient Multi Hop Retrieval Augmented Generation
  8. 2026Understanding the Behaviors of Environment-aware Information Retrieval