mlsanityData to 14 Oct 2026

LeTS: Learning to Think-and-Search via Process-and-Outcome Reward Hybridization

arXiv 2505.17447 ↗First posted 23 May 2025cs.CLcited by 10

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive capabilities in reasoning with the emergence of reasoning models like OpenAI-o1 and DeepSeek-R1. Recent research focuses on integrating reasoning capabilities into the realm of retrieval-augmented generation (RAG) via outcome-supervised reinforcement learning (RL) approaches, while the correctness of intermediate think-and-search steps is usually neglected. To address this issue, we design a process-level reward module to mitigate the unawareness of intermediate reasoning steps in outcome-level supervision without additional annotation. Grounded on this, we propose Learning to Think-and-Search (LeTS), a novel framework that hybridizes stepwise process reward and outcome-based reward to current RL methods for RAG. Extensive experiments demonstrate the generalization and inference efficiency of LeTS across various RAG benchmarks. In addition, these results reveal the potential of process- and outcome-level reward hybridization in boosting LLMs' reasoning ability via RL under other scenarios. The code will be released soon.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 39 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

19 of 39 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning25 Mar
  2. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning12 Mar
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  4. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions20 Dec

Other references · 15

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2023A Survey of Large Language Modelscited by 2,576
  6. 2024OpenAI o1 System Cardcited by 1,244
  7. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  8. 2024A Survey on Large Language Models for Code Generationcited by 658
  9. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  10. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 346
  11. 2024RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generationcited by 147
  12. 2025A Survey of Large Language Model Agents for Question Answeringcited by 48
3 more
  1. 2024RA-ISF: Learning to Answer and Understand from Retrieval Augmentation via Iterative Self-Feedbackcited by 46
  2. 2023Understanding Retrieval Augmentation for Long-Form Question Answeringcited by 45
  3. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 39

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 7.

10 papers.

  1. 2025CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Criticcited by 30
  2. 2025From Sufficiency to Reflection: Reinforcement-Guided Thinking Quality in Retrieval-Augmented Reasoning for LLMscited by 5
  3. 2025GlobalRAG: Enhancing Global Reasoning in Multi-hop Question Answering via Reinforcement Learningcited by 5
  4. 2026Agentic Tool Use in Large Language Modelscited by 4
  5. 2026ARBOR: Online Process Rewards via a Reusable Rubric Buffer for Search Agentscited by 3
  6. 2026Self-Manager: Parallel Agent Loop for Long-form Deep Researchcited by 1
  7. 2026EviBack: Search-Agent Reinforcement Learning via Evidence-Constrained Teacher Backoffcited by 0
  8. 2026Cross-View Correspondence Is a Measurement Intervention: Two-Sided Validation for Agent Evaluation and Credit Assignmentcited by 0
  9. 2026MCite-RL: Towards Reliable Multimodal RAG via Citation-enhanced Agentic Reinforcement Learningcited by 0
  10. 2026Beyond Dyadic Memory: Interaction-Aware Multimodal Memory with Adaptive Agentic Retrieval for Multi-Party Spoken Conversationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Thinking Forward and Backward: Multi-Objective Reinforcement Learning for Retrieval-Augmented Reasoning
  2. 2026Search-P1: Path-Centric Reward Shaping for Stable and Efficient Agentic RAG Training
  3. 2025ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewarding
  4. 2026ProRAG: Process-Supervised Reinforcement Learning for Retrieval-Augmented Generation
  5. 2025RouteRAG: Efficient Retrieval-Augmented Generation from Text and Graph via Reinforcement Learning
  6. 2025UR2: Unify RAG and Reasoning through Reinforcement Learning
  7. 2025Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning
  8. 2026PRO-Step: Step-level Process Reward Optimization for Retrieval-Augmented Generation