mlsanityData to 7 Oct 2026

Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoning

arXiv 2505.11277 ↗First posted 16 May 2025cs.CL cs.AIcited by 34

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

also in 1 more
Abstract (arXiv)

Large language models have demonstrated impressive reasoning capabilities but are inherently limited by their knowledge reservoir. Retrieval-augmented reasoning mitigates this limitation by allowing LLMs to query external resources, but existing methods often retrieve irrelevant or noisy information, hindering accurate reasoning. In this paper, we propose AutoRefine, a reinforcement learning post-training framework that adopts a new "search-and-refine-during-think" paradigm. AutoRefine introduces explicit knowledge refinement steps between successive search calls, enabling the model to iteratively filter, distill, and organize evidence before generating an answer. Furthermore, we incorporate tailored retrieval-specific rewards alongside answer correctness rewards using group relative policy optimization. Experiments on single-hop and multi-hop QA benchmarks demonstrate that AutoRefine significantly outperforms existing approaches, particularly in complex, multi-hop reasoning scenarios. Detailed analysis shows that AutoRefine issues frequent, higher-quality searches and synthesizes evidence effectively.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 39 of its 72 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

39 of 72 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025RARE: Retrieval-Augmented Reasoning Modeling30 Mar
  2. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning25 Mar
  3. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning12 Mar
  4. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning7 Mar
  5. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Models9 Jan
  6. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec

Other references · 33

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2024Qwen2.5 Technical Reportcited by 4,713
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  9. 2024OpenAI o1 System Cardcited by 1,220
  10. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  11. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  12. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
21 more
  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  3. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
  4. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 268
  5. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalizationcited by 233
  6. 2024Corrective Retrieval Augmented Generationcited by 187
  7. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  8. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 137
  9. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplificationcited by 108
  10. 2025Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learningcited by 107
  11. 2025Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning with Agentic Toolscited by 106
  12. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 103
  13. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Modelscited by 81
  14. 2024o1-Coder: an o1 Replication for Codingcited by 77
  15. 2024Interpretable Contrastive Monte Carlo Tree Search Reasoningcited by 62
  16. 2025DeepRetrieval: Hacking Real Search Engines and Retrievers with Large Language Models via Reinforcement Learningcited by 62
  17. 2025Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learningcited by 39
  18. 2025RAG-RL: Advancing Retrieval-Augmented Generation via RL and Curriculum Learningcited by 30
  19. 2025Beyond Outlining: Heterogeneous Recursive Planning for Adaptive Long-form Writing with Language Modelscited by 15
  20. 2025DeepSolution: Boosting Complex Engineering Solution Design via Tree-based Exploration and Bi-point Thinkingcited by 9
  21. 2025SEM: Reinforcement Learning for Search-Efficient Large Language Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 24.

34 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  2. 2025Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Modelscited by 36
  3. 2025LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contextscited by 25
  4. 2026MemOCR: Layout-Aware Visual Memory for Efficient Long-Horizon Reasoningcited by 15
  5. 2026Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoningcited by 9
  6. 2026When to Memorize and When to Stop: Gated Recurrent Memory for Long-Context Reasoningcited by 8
  7. 2026IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoningcited by 8
  8. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  9. 2026OASES: Outcome-Aligned Search-Evaluation Co-Training for Agentic Searchcited by 7
  10. 2025Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generationcited by 6
  11. 2026Optimizing Agentic Reasoning with Retrieval via Synthetic Semantic Information Gain Rewardcited by 6
  12. 2025From Sufficiency to Reflection: Reinforcement-Guided Thinking Quality in Retrieval-Augmented Reasoning for LLMscited by 5
22 more
  1. 2026SD-Search: On-Policy Hindsight Self-Distillation for Search-Augmented Reasoningcited by 5
  2. 2025GlobalRAG: Enhancing Global Reasoning in Multi-hop Question Answering via Reinforcement Learningcited by 4
  3. 2026MemSearch-o1: Empowering Large Language Models with Reasoning-Aligned Memory Growth in Agentic Searchcited by 4
  4. 2026SIGHT: Reinforcement Learning with Self-Evidence and Information-Gain Diverse Branching for Search Agentcited by 3
  5. 2026EIBench: A Simulator-Based Benchmark and Turn-Credit RL for Emotion Managementcited by 3
  6. 2026ORBIT: Scalable and Verifiable Data Generation for Search Agents on a Tight Budgetcited by 2
  7. 2026PiCA: Pivot-Based Credit Assignment for Search Agentic Reinforcement Learningcited by 2
  8. 2026SearchMaster: Grounded and Regulated Self-Play for Search Agentscited by 2
  9. 2025Plan Then Retrieve: Reinforcement Learning-Guided Complex Reasoning over Knowledge Graphscited by 1
  10. 2026TKG-Thinker: Towards Dynamic Reasoning over Temporal Knowledge Graphs via Agentic Reinforcement Learningcited by 1
  11. 2026A Picture of Agentic Searchcited by 1
  12. 2026Latent Abstraction for Retrieval-Augmented Generationcited by 1
  13. 2026Track, Rank, Crack: Epistemic Working Memory Scales Multi-Hop Reasoning in Language Agentscited by 1
  14. 2025Thinking Forward and Backward: Multi-Objective Reinforcement Learning for Retrieval-Augmented Reasoningcited by 0
  15. 2025HuggingR4: A Progressive Reasoning Framework for Discovering Optimal Model Companionscited by 0
  16. 2025Laser: Governing Long-Horizon Agentic Search via Structured Protocol and Context Registercited by 0
  17. 2026Scaling Retrieval-Augmented Reasoning with Parallel Search and Explicit Mergingcited by 0
  18. 2026CAPF: Guiding Search-Agent Rollouts with Credit-Attenuated Privileged Feedbackcited by 0
  19. 2026R2-Searcher: Calibrating Retrieval and Reasoning Boundaries for Agentic Searchcited by 0
  20. 2026DeepStress: Stress-Testing Deep Search Agentscited by 0
  21. 2026HindSearch: Trajectory-Level Hindsight Critique for Search-Augmented Reinforcement Learningcited by 0
  22. 2026IGSD: Environment-Verified Hindsight Self-Distillation for Search Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ReaRAG: Knowledge-guided Reasoning Enhances Factuality of Large Reasoning Models with Iterative Retrieval Augmented Generation
  2. 2026One Refiner to Unlock Them All: Inference-Time Reasoning Elicitation via Reinforcement Query Refinement
  3. 2026OThink-SRR1: Search, Refine and Reasoning with Reinforced Learning for Large Language Models
  4. 2026Learning to Reason by Analogy via Retrieval-Augmented Reinforcement Fine-Tuning
  5. 2026D2Plan: Dual-Agent Dynamic Global Planning for Complex Retrieval-Augmented Reasoning
  6. 2025Reinforcement Fine-Tuning for Reasoning towards Multi-Step Multi-Source Search in Large Language Models
  7. 2025R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning
  8. 2025UR2: Unify RAG and Reasoning through Reinforcement Learning