mlsanityData to 14 Oct 2026

SOAR: Source-Aware Reinforcement Learning with Precise Credit Assignment for Search Agents

arXiv 2610.13157 ↗First posted 9 Oct 2026cs.CLcited by 0

On the map: Deep reinforcement learning methods → RL training of language and agent models also fits: Interpretable and logic-guided RL

Abstract (arXiv)

Reinforcement learning (RL) for search agents often suffers from evidence fabrication, a failure mode where agents bypass external information (Eext) and fabricate an incorrect answer. We argue that this fabrication arises from an over-reliance on unreliable internal knowledge (Eint) as shortcuts due to the inherent opacity of reasoning trajectories. To address this, we propose SOAR (SOurce-Aware Reinforcement learning), a paradigm that transforms monolithic reasoning into source-explicit trajectories. SOAR (i) disentangles Eext and Eint into parallel, observable streams via a Source-Aware Workflow, and (ii) incorporates Source-Consistent Reward to guide the policy evolution. By making information-source attribution explicit, SOAR enables precise credit assignment between different knowledge streams. Evaluations show that SOAR outperforms baselines on 3B and 7B backbones by 7.3% and 5.4%, while reducing evidence fabrication by 74.8%. These confirm that SOAR significantly improves performance on knowledge-intensive benchmarks while effectively mitigating evidence fabrication, fostering more faithful and grounded reasoning with enhanced robustness.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 30 of its 50 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

30 of 50 references.

References · 30

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  9. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  10. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  11. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  12. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
18 more
  1. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  2. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 234
  3. 2023On Early Detection of Hallucinations in Factual Question Answeringcited by 56
  4. 2018Reinforcement Learning to Rank in E-Commerce Search Engine: Formalization, Analysis, and Applicationcited by 34
  5. 2025Reinforced Internal-External Knowledge Synergistic Reasoning for Efficient Adaptive Search Agentcited by 30
  6. 2026SmartSearch: Process Reward-Guided Query Refinement for Search Agentscited by 14
  7. 2025ReAgent: Reversible Multi-Agent Reasoning for Knowledge-Enhanced Multi-Hop QAcited by 13
  8. 2026Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaborationcited by 11
  9. 2024Face4RAG: Factual Consistency Evaluation for Retrieval Augmented Generation in Chinesecited by 10
  10. 2025SPARK: Synergistic Policy And Reward Co-Evolving Frameworkcited by 10
  11. 2026IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoningcited by 9
  12. 2025What Is Your Agent's GPA? A Framework for Evaluating Agent Goal-Plan-Action Alignmentcited by 8
  13. 2026How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1cited by 8
  14. 2025FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoningcited by 7
  15. 2025AdaSearch: Balancing Parametric Knowledge and Search in Large Language Models via Reinforcement Learningcited by 6
  16. 2026Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoningcited by 6
  17. 2026EvidenceRL: Reinforcing Evidence Consistency for Trustworthy Language Modelscited by 5
  18. 2026Dep-Search: Learning Dependency-Aware Reasoning Traces with Persistent Memorycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewards
  2. 2026Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents
  3. 2026Contextual Information Policy Optimization for Search Agents
  4. 2026Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards
  5. 2026SD-Search: On-Policy Hindsight Self-Distillation for Search-Augmented Reasoning
  6. 2026BAPO: Boundary-Aware Policy Optimization for Reliable Agentic Search
  7. 2025ReLaX: Reasoning with Latent Exploration for Large Reasoning Models
  8. 2026SAAS: Self-Aware Reinforcement Learning for Over-Search Mitigation in Agentic Search