# ExploRLLM: Guiding Exploration in Reinforcement Learning with Large Language Models

arXiv: https://arxiv.org/abs/2403.09583 · HTML: https://mlsanity.com/paper/2403.09583/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Mar 2024.

## Built on by (most cited first, 19 of 19)

- [Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning](https://mlsanity.com/paper/2605.02913/) (2026)
- [Knowledge Integration for Physics-informed Symbolic Regression Using Pre-trained Large Language Models](https://mlsanity.com/paper/2509.03036/) (2025)
- [Zero-shot Model-based Reinforcement Learning using Large Language Models](https://mlsanity.com/paper/2410.11711/) (2024)
- [Swarms of Large Language Model Agents for Protein Sequence Design with Experimental Validation](https://mlsanity.com/paper/2511.22311/) (2025)
- [Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videos](https://mlsanity.com/paper/2410.09286/) (2024)
- [MISR: Measuring Instrumental Self-Reasoning in Frontier Models](https://mlsanity.com/paper/2412.03904/) (2024)
- [Towards Autonomous Reinforcement Learning for Real-World Robotic Manipulation with Large Language Models](https://mlsanity.com/paper/2503.04280/) (2025)
- [LLM-based Interactive Imitation Learning for Robotic Manipulation](https://mlsanity.com/paper/2504.21769/) (2025)
- [APEX: Empowering LLMs with Physics-Based Task Planning for Real-time Insight](https://mlsanity.com/paper/2505.13921/) (2025)
- [Population-Coded Spiking Neural Networks for High-Dimensional Robotic Control](https://mlsanity.com/paper/2510.10516/) (2025)
- [An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problems](https://mlsanity.com/paper/2501.00562/) (2024)
- [Towards a Reward-Free Reinforcement Learning Framework for Vehicle Control](https://mlsanity.com/paper/2502.15262/) (2025)
- [Accelerating Reinforcement Learning Algorithms Convergence using Pre-trained Large Language Models as Tutors With Advice Reusing](https://mlsanity.com/paper/2509.08329/) (2025)
- [Curriculum-Based Multi-Tier Semantic Exploration via Deep Reinforcement Learning](https://mlsanity.com/paper/2509.09356/) (2025)
- [Agent2: An Agent-Generates-Agent Framework for Reinforcement Learning Automation](https://mlsanity.com/paper/2509.13368/) (2025)
- [VLM-Guided Experience Replay](https://mlsanity.com/paper/2602.01915/) (2026)
- [Efficient Soft Actor-Critic with LLM-Based Action-Level Guidance for Continuous Control](https://mlsanity.com/paper/2603.17468/) (2026)
- [Reinforcement Learning from Cross-domain Videos with Video Prediction Model](https://mlsanity.com/paper/2606.03201/) (2026)
- [Prompt-Driven Exploration: Language as an Exploration Space for VLA Reinforcement Learning](https://mlsanity.com/paper/2607.08837/) (2026)

## Abstract (arXiv)

In robot manipulation, Reinforcement Learning (RL) often suffers from low sample efficiency and uncertain convergence, especially in large observation and action spaces. Foundation Models (FMs) offer an alternative, demonstrating promise in zero-shot and few-shot settings. However, they can be unreliable due to limited physical and spatial understanding. We introduce ExploRLLM, a method that combines the strengths of both paradigms. In our approach, FMs improve RL convergence by generating policy code and efficient representations, while a residual RL agent compensates for the FMs' limited physical understanding. We show that ExploRLLM outperforms both policies derived from FMs and RL baselines in table-top manipulation tasks. Additionally, real-world experiments show that the policies exhibit promising zero-shot sim-to-real transfer. Supplementary material is available at https://explorllm.github.io.