mlsanityData to 7 Oct 2026

Toward Efficient Exploration by Large Language Model Agents

arXiv 2504.20997 ↗First posted 29 Apr 2025cs.LG cs.AIcited by 15

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

A burgeoning area within reinforcement learning (RL) is the design of sequential decision-making agents centered around large language models (LLMs). While autonomous decision-making agents powered by modern LLMs could facilitate numerous real-world applications, such successes demand agents that are capable of data-efficient RL. One key obstacle to achieving data efficiency in RL is exploration, a challenge that we demonstrate many recent proposals for LLM agent designs struggle to contend with. Meanwhile, classic algorithms from the RL literature known to gracefully address exploration require technical machinery that can be challenging to operationalize in purely natural language settings. In this work, rather than relying on finetuning or in-context learning to coax LLMs into implicitly imitating a RL algorithm, we illustrate how LLMs can be used to explicitly implement an existing RL algorithm (Posterior Sampling for Reinforcement Learning) whose capacity for statistically-efficient exploration is already well-studied. We offer empirical results demonstrating how our LLM-based implementation of a known, data-efficient RL algorithm can be considerably more effective in natural language tasks that demand prudent exploration.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 79 of its 123 references that are in the map. First posted on arXiv 29 Apr 2025.

Builds on

79 of 123 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedback30 Oct
  2. 2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models19 May
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 76

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
38 more
  1. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  2. 2016Deep Exploration via Bootstrapped DQNcited by 518
  3. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  4. 2019Model-Based Reinforcement Learning for Ataricited by 495
  5. 2018Is Q-learning Provably Efficient?cited by 485
  6. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  7. 2017A Tutorial on Thompson Samplingcited by 420
  8. 2022Using cognitive psychology to understand GPT-3cited by 389
  9. 2023Reward Design with Language Modelscited by 257
  10. 2013Learning to Optimize Via Posterior Samplingcited by 253
  11. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  12. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  13. 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 194
  14. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
  15. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  16. 2014An Information-Theoretic Analysis of Thompson Samplingcited by 178
  17. 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 175
  18. 2014Learning to Optimize via Information-Directed Samplingcited by 168
  19. 2017Deep Exploration via Randomized Value Functionscited by 155
  20. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  21. 2014Model-based Reinforcement Learning and the Eluder Dimensioncited by 116
  22. 2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 116
  23. 2014Generalization and Exploration via Randomized Value Functionscited by 112
  24. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  25. 2015Sample Complexity of Episodic Fixed-Horizon Reinforcement Learningcited by 105
  26. 2015Contextual Dueling Banditscited by 101
  27. 2017The Uncertainty Bellman Equation and Explorationcited by 81
  28. 2024Can large language models explore in-context?cited by 75
  29. 2017Learning Unknown Markov Decision Processes: A Thompson Sampling Approachcited by 65
  30. 2017Ensemble Samplingcited by 62
  31. 2024CogBench: a large language model walks into a psychology labcited by 60
  32. 2021Reinforcement Learning, Bit by Bitcited by 59
  33. 2023Meta-in-context learning in large language modelscited by 59
  34. 2019Information-Theoretic Confidence Bounds for Reinforcement Learningcited by 51
  35. 2023Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiencycited by 46
  36. 2024Efficient Exploration for LLMscited by 44
  37. 2021Bayesian decision-making under misspecified priors with applications to meta-learningcited by 43
  38. 2021Simple Agent, Complex Environment: Efficient Reinforcement Learning with Agent Statescited by 42

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 6.

15 papers.

Sort:

By area:

  1. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  2. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  3. 2025Are Large Language Models Reliable AI Scientists? Assessing Reverse-Engineering of Black-Box Systemscited by 13
  4. 2025Formalizing Learning from Language Feedback with Provable Guaranteescited by 10
  5. 2025Beyond Markovian: Reflective Exploration via Bayes-Adaptive RL for LLM Reasoningcited by 8
  6. 2025Dynamic Risk Assessments for Offensive Cybersecurity Agentscited by 7
  7. 2025ABBEL: Learning Natural-Language Belief States for Memory-Efficient Interactioncited by 7
  8. 2025CaRT: Teaching LLM Agents to Know When They Know Enoughcited by 5
  9. 2025Improving the Data-efficiency of Reinforcement Learning by Warm-starting with LLMcited by 2
  10. 2026Cognitive Models and AI Algorithms Provide Templates for Designing Language Agentscited by 2
  11. 2026A Component-Based Survey of Interactions between Large Language Models and Multi-Armed Banditscited by 1
  12. 2026Reasonably reasoning AI agents can avoid game-theoretic failures in zero-shot, provablycited by 1
3 more
  1. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutionscited by 0
  2. 2026Agent2UCB: Agentic System for Generative Engine Optimizationcited by 0
  3. 2026Success Leaves Detours: Learning Executable Walkthroughs for Long-Horizon Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models
  2. 2025Guiding Exploration in Reinforcement Learning Through LLM-Augmented Observations
  3. 2025Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs
  4. 2025Is a Good Foundation Necessary for Efficient Reinforcement Learning? The Computational Role of the Base Model in Exploration
  5. 2024Language Guided Exploration for RL Agents in Text Environments
  6. 2025Improving RL Exploration for LLM Reasoning through Retrospective Replay
  7. 2025The Road Less Traveled: Enhancing Exploration in LLMs via Sequential Sampling
  8. 2026Rethinking Agentic Reinforcement Learning In Large Language Models