mlsanityData to 14 Oct 2026

Goal-Guided Efficient Exploration via Large Language Model in Reinforcement Learning

arXiv 2509.22008 ↗First posted 26 Sep 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Real-world decision-making tasks typically occur in complex and open environments, posing significant challenges to reinforcement learning (RL) agents' exploration efficiency and long-horizon planning capabilities. A promising approach is LLM-enhanced RL, which leverages the rich prior knowledge and strong planning capabilities of LLMs to guide RL agents in efficient exploration. However, existing methods mostly rely on frequent and costly LLM invocations and suffer from limited performance due to the semantic mismatch. In this paper, we introduce a Structured Goal-guided Reinforcement Learning (SGRL) method that integrates a structured goal planner and a goal-conditioned action pruner to guide RL agents toward efficient exploration. Specifically, the structured goal planner utilizes LLMs to generate a reusable, structured function for goal generation, in which goals are prioritized. Furthermore, by utilizing LLMs to determine goals' priority weights, it dynamically generates forward-looking goals to guide the agent's policy toward more promising decision-making trajectories. The goal-conditioned action pruner employs an action masking mechanism that filters out actions misaligned with the current goal, thereby constraining the RL agent to select goal-consistent policies. We evaluate the proposed method on Crafter and Craftax-Classic, and experimental results demonstrate that SGRL achieves superior performance compared to existing state-of-the-art methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 36 of its 45 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

36 of 45 references.

References · 36

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  6. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  7. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  8. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  9. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  10. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  11. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 224
  12. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 200
24 more
  1. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 195
  2. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 187
  3. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 182
  4. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 158
  5. 2023Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learningcited by 133
  6. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 88
  7. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 86
  8. 2023Language Instructed Reinforcement Learning for Human-AI Coordinationcited by 63
  9. 2023Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Predictioncited by 39
  10. 2024Efficient Reinforcement Learning with Large Language Model Priorscited by 30
  11. 2021JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learningcited by 28
  12. 2023Enabling Intelligent Interactions between an Agent and an LLM: A Reinforcement Learning Approachcited by 28
  13. 2024From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systemscited by 17
  14. 2025Automated Skill Discovery for Language Agents through Exploration and Iterative Feedbackcited by 12
  15. 2023LgTS: Dynamic Task Sampling using LLM-generated sub-goals for Reinforcement Learning Agentscited by 11
  16. 2023AdaRefiner: Refining Decisions of Language Models with Adaptive Feedbackcited by 10
  17. 2024Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedbackcited by 10
  18. 2025The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learningcited by 10
  19. 2020Dialog Policy Learning for Joint Clarification and Active Learning Queriescited by 9
  20. 2023LLM Augmented Hierarchical Agentscited by 9
  21. 2023Discovering Hierarchical Achievements in Reinforcement Learning via Contrastive Learningcited by 6
  22. 2023Investigating the role of model-based learning in exploration and transfercited by 5
  23. 2024SAC-GLAM: Improving Online RL for LLM agents with Soft Actor-Critic and Hindsight Relabelingcited by 5
  24. 2024Words as Beacons: Guiding RL Agents with High-Level Language Promptscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Guiding Exploration in Reinforcement Learning Through LLM-Augmented Observations
  2. 2025LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models
  3. 2025Toward Efficient Exploration by Large Language Model Agents
  4. 2026Expanding LLM Agent Boundaries with Strategy-Guided Exploration
  5. 2025Can LLMs Guide Their Own Exploration? Gradient-Guided Reinforcement Learning for LLM Reasoning
  6. 2024Reinforcement Learning from LLM Feedback to Counteract Goal Misgeneralization
  7. 2025Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
  8. 2024Stop Wandering, Find the Keys: LLMs Discriminate Key States for Efficient Multi-Agent Exploration