mlsanityData to 14 Oct 2026

Meta-Reinforcement Learning with Self-Reflection for Agentic Search

arXiv 2603.11327 ↗First posted 11 Mar 2026cs.LG cs.CLcited by 8

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

This paper introduces MR-Search, an in-context meta reinforcement learning (RL) formulation for agentic search with self-reflection. Instead of optimizing a policy within a single independent episode with sparse rewards, MR-Search trains a policy that conditions on past episodes and adapts its search strategy across episodes. MR-Search learns to learn a search strategy with self-reflection, allowing search agents to improve in-context exploration at test-time. Specifically, MR-Search performs cross-episode exploration by generating explicit self-reflections after each episode and leveraging them as additional context to guide subsequent attempts, thereby promoting more effective exploration during test-time. We further introduce a multi-turn RL algorithm that estimates a dense relative advantage at the turn level, enabling fine-grained credit assignment on each episode. Empirical results across various benchmarks demonstrate the advantages of MR-Search over baselines based RL, showing strong generalization and relative improvements of 9.2% to 19.3% across eight benchmarks. Our code and data are available at https://github.com/tengxiao1/MR-Search.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 53 of its 64 references that are in the map. First posted on arXiv 11 Mar 2026.

Builds on

53 of 64 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Meta-RL Induces Exploration in Language Agents18 Dec
  2. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 50

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  10. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  11. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  12. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
38 more
  1. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  2. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  3. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  4. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  5. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  6. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  7. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  8. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  9. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 433
  10. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  11. 2016Learning to reinforcement learncited by 377
  12. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 336
  13. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 292
  14. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 286
  15. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 248
  16. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 229
  17. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 208
  18. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  19. 2025WebDancer: Towards Autonomous Information Seeking Agencycited by 188
  20. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 165
  21. 2024Free Process Rewards without Process Labelscited by 165
  22. 2025Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RLcited by 150
  23. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 123
  24. 2026Learning to Discover at Test Timecited by 115
  25. 2025DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Researchcited by 114
  26. 2025StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimizationcited by 88
  27. 2025Agent Learning via Early Experiencecited by 85
  28. 2025Self-rewarding correction for mathematical reasoningcited by 70
  29. 2020Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrificescited by 59
  30. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learningcited by 57
  31. 2018Some Considerations on Learning to Explore via Meta-Reinforcement Learningcited by 44
  32. 2022Transformers are Meta-Reinforcement Learnerscited by 41
  33. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  34. 2026Towards Execution-Grounded Automated AI Researchcited by 28
  35. 2025Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought Rewardcited by 25
  36. 2025Uncalibrated Reasoning: GRPO Induces Overconfidence for Stochastic Outcomescited by 22
  37. 2025Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Modelscited by 18
  38. 2025SSRL: Self-Search Reinforcement Learningcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Agentic Aggregation for Parallel Scaling of Long-Horizon Agentic Taskscited by 18
  2. 2026IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoningcited by 9
  3. 2026AdaMEM: Test-Time Adaptive Memory for Language Agentscited by 6
  4. 2026Procedural Memory Distillation: Online Reflection for Self-Improving Language Modelscited by 6
  5. 2026SD-Search: On-Policy Hindsight Self-Distillation for Search-Augmented Reasoningcited by 5
  6. 2026Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retrycited by 3
  7. 2026Connect the Dots: Training LLMs for Long-Lifecycle Agents with Cross-Domain Generalization Via Reinforcement Learningcited by 0
  8. 2026Co-Evolving Graph and Text Memory for Training-Free Multi-Hop Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications
  2. 2026MAGE: Meta-Reinforcement Learning for Language Agents toward Strategic Exploration and Exploitation
  3. 2024MAMBA: an Effective World Model Approach for Meta-Reinforcement Learning
  4. 2025WebSeer: Training Deeper Search Agents through Reinforcement Learning with Self-Reflection
  5. 2024DynaMITE-RL: A Dynamic Model for Improved Temporal Meta-Reinforcement Learning
  6. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers
  7. 2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning
  8. 2024Theoretical Analysis of Meta Reinforcement Learning: Generalization Bounds and Convergence Guarantees