mlsanityData to 14 Oct 2026

Exploration with Foundation Models: Capabilities, Limitations, and Hybrid Approaches

arXiv 2509.19924 ↗First posted 24 Sep 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Exploration in reinforcement learning (RL) remains challenging, particularly in sparse-reward settings. While foundation models possess strong semantic priors, their capabilities as zero-shot exploration agents in classic RL benchmarks are not well understood. We benchmark LLMs and VLMs on multi-armed bandits, Gridworlds, and sparse-reward Atari to test zero-shot exploration. Our investigation reveals a key limitation: while VLMs can infer high-level objectives from visual input, they consistently fail at precise low-level control: the "knowing-doing gap". To analyze a potential bridge for this gap, we investigate a simple on-policy hybrid framework in a controlled, best-case scenario. Our results in this idealized setting show that VLM guidance can significantly improve early-stage sample efficiency, providing a clear analysis of the potential and constraints of using foundation models to guide exploration rather than for end-to-end control.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 28 references that are in the map. First posted on arXiv 24 Sep 2025.

Builds on

22 of 28 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  6. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  7. 2024Mixtral of Expertscited by 1,823
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  9. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  10. 2018Exploration by Random Network Distillationcited by 993
  11. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  12. 2023Large Language Modelscited by 780
10 more
  1. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  2. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 187
  3. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 146
  4. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 125
  5. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
  6. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 102
  7. 2024Intelligent Go-Explore: Standing on the Shoulders of Giant Foundation Modelscited by 33
  8. 2023Posterior Sampling for Deep Reinforcement Learningcited by 16
  9. 2025TextAtari: 100K Frames Game Playing with Language Agentscited by 6
  10. 2025Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Makingcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning
  2. 2024Fostering Intrinsic Motivation in Reinforcement Learning with Pretrained Foundation Models
  3. 2019Go-Explore: a New Approach for Hard-Exploration Problems
  4. 2026Search Inspired Exploration in Reinforcement Learning
  5. 2025Guiding Exploration in Reinforcement Learning Through LLM-Augmented Observations
  6. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
  7. 2026Parameter Exploration for RLVR via Variational Learning
  8. 2026Decoupling Exploration from Optimization in RLVR