mlsanityData to 13 Oct 2026

Meta-RL Induces Exploration in Language Agents

arXiv 2512.16848 ↗First posted 18 Dec 2025cs.LG cs.AIcited by 20

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

also in 1 more
Abstract (arXiv)

Reinforcement learning (RL) has enabled the training of large language model (LLM) agents to interact with the environment and to solve multi-turn long-horizon tasks. However, the RL-trained agents often struggle in tasks that require active exploration and fail to efficiently adapt from trial-and-error experiences. In this paper, we present LaMer, a general Meta-RL framework that enables LLM agents to actively explore and learn from the environment feedback at test time. LaMer consists of two key components: (i) a cross-episode training framework to encourage exploration and long-term rewards optimization; and (ii) in-context policy adaptation via reflection, allowing the agent to adapt their policy from task feedback signal without gradient update. Experiments across diverse environments show that LaMer significantly improves performance over RL baselines, with 11%, 14%, and 19% performance gains on Sokoban, MineSweeper and Webshop, respectively. Moreover, LaMer also demonstrates better generalization to more challenging or previously unseen tasks compared to the RL-trained agents. Overall, our results demonstrate that Meta-RL provides a principled approach to induce exploration in language agents, enabling more robust adaptation to novel environments through learned exploration strategies.

Key facts. Cited by 20 papers in the map (data through 13 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 46 of its 57 references that are in the map. First posted on arXiv 18 Dec 2025.

Builds on

46 of 57 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Survey2 Sep
  2. 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games4 Jun
  3. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  4. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2017Proximal Policy Optimization Algorithmscited by 14,592
  3. 2024The Llama 3 Herd of Modelscited by 14,486
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,091
  7. 2025Qwen3 Technical Reportcited by 8,510
  8. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,502
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,270
  10. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,870
  11. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,541
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,181
28 more
  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,218
  2. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,003
  3. 2025s1: Simple test-time scalingcited by 1,322
  4. 2023AgentBench: Evaluating LLMs as Agentscited by 1,242
  5. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,219
  6. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,167
  7. 2020Meta-Learning in Neural Networks: A Surveycited by 945
  8. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 913
  9. 2021Measuring Coding Challenge Competence With APPScited by 891
  10. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 494
  11. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 413
  12. 2016Learning to reinforcement learncited by 373
  13. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 237
  14. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  15. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 164
  16. 2024Stream of Search (SoS): Learning to Search in Languagecited by 142
  17. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 138
  18. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  19. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 122
  20. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 118
  21. 2023A Tutorial on Meta-Reinforcement Learningcited by 104
  22. 2024Can large language models explore in-context?cited by 75
  23. 2025Self-rewarding correction for mathematical reasoningcited by 69
  24. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 57
  25. 2018Some Considerations on Learning to Explore via Meta-Reinforcement Learningcited by 44
  26. 2024EVOLvE: Evaluating and Optimizing LLMs For In-Context Explorationcited by 34
  27. 2025Training a Generally Curious Agentcited by 30
  28. 2023Assessing Logical Puzzle Solving in Large Language Models: Insights from a Minesweeper Case Studycited by 14

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 20.

20 papers.

Sort:

By area:

  1. 2026Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learningcited by 34
  2. 2026RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedbackcited by 16
  3. 2026Meta-Reinforcement Learning with Self-Reflection for Agentic Searchcited by 8
  4. 2026Test-time Recursive Thinking: Self-Improvement without External Feedbackcited by 7
  5. 2026AdaMEM: Test-Time Adaptive Memory for Language Agentscited by 5
  6. 2026SkillRise: Agentic Reinforcement Learning for Cross-Task Skill Evolutioncited by 4
  7. 2026Meta-TTL: Meta-Learning Self-Improvement Policies for Language Agentscited by 3
  8. 2026Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retrycited by 3
  9. 2026MAGE: Meta-Reinforcement Learning for Language Agents toward Strategic Exploration and Exploitationcited by 2
  10. 2026LatentGym: A Testbed For Cross-Task Experiential Learning With Controllable Latent Structurecited by 2
  11. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  12. 2026Training Language Agents to Learn from Experiencecited by 1
8 more
  1. 2026Three Roles, One Model: Role Orchestration at Inference Time to Close the Performance Gap Between Small and Large Agentscited by 0
  2. 2026Mags-RL: Wearing Multimodal LLMs a Magnifying Glass via Agentic Reinforcement Learning For Complex Scene Reasoningcited by 0
  3. 2026Connect the Dots: Training LLMs for Long-Lifecycle Agents with Cross-Domain Generalization Via Reinforcement Learningcited by 0
  4. 2026Act More, Decide Less: Skill-Guided Adaptive Action Chunking for Long-Horizon LLM Agentscited by 0
  5. 2026CoSkill: Joint Reinforcement Learning of Reasoning and Meta-Skill Agents for Hierarchical Skill Evolutioncited by 0
  6. 2026Success Leaves Detours: Learning Executable Walkthroughs for Long-Horizon Agentscited by 0
  7. 2026From Memory to Guide: Spatio-Temporal Composer for Procedural Coding Memorycited by 0
  8. 2026FC-SWE: Failure-Conditioned RL for Long-Horizon Software Engineering Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Training
  2. 2024Words as Beacons: Guiding RL Agents with High-Level Language Prompts
  3. 2025Toward Efficient Exploration by Large Language Model Agents
  4. 2024Language Guided Exploration for RL Agents in Text Environments
  5. 2025Improving RL Exploration for LLM Reasoning through Retrospective Replay
  6. 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments
  7. 2025Guiding Exploration in Reinforcement Learning Through LLM-Augmented Observations
  8. 2024Mutual Enhancement of Large Language and Reinforcement Learning Models through Bi-Directional Feedback Mechanisms: A Planning Case Study