# Game On: Towards Language Models as RL Experimenters

arXiv: https://arxiv.org/abs/2409.03402 · HTML: https://mlsanity.com/paper/2409.03402/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 43 references that are in the map.
- First posted on arXiv 5 Sep 2024.

## Builds on (most cited first, 29 of 29)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos](https://mlsanity.com/paper/2206.11795/) (2022)
- [Maximum a Posteriori Policy Optimisation](https://mlsanity.com/paper/1806.06920/) (2018)
- [Language to Rewards for Robotic Skill Synthesis](https://mlsanity.com/paper/2306.08647/) (2023)
- [SciMON: Scientific Inspiration Machines Optimized for Novelty](https://mlsanity.com/paper/2305.14259/) (2023)
- [JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models](https://mlsanity.com/paper/2311.05997/) (2023)
- [Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning](https://mlsanity.com/paper/2310.12921/) (2023)
- [PaperQA: Retrieval-Augmented Generative Agent for Scientific Research](https://mlsanity.com/paper/2312.07559/) (2023)
- [Vision-Language Models as Success Detectors](https://mlsanity.com/paper/2303.07280/) (2023)
- [GenSim: Generating Robotic Simulation Tasks via Large Language Models](https://mlsanity.com/paper/2310.01361/) (2023)
- [RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation](https://mlsanity.com/paper/2306.11706/) (2023)
- [STEVE-1: A Generative Model for Text-to-Behavior in Minecraft](https://mlsanity.com/paper/2306.00937/) (2023)
- [Execution-Based Evaluation for Open-Domain Code Generation](https://mlsanity.com/paper/2212.10481/) (2022)
- [Natural Language to Code Generation in Interactive Data Science Notebooks](https://mlsanity.com/paper/2212.09248/) (2022)
- [LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discovery](https://mlsanity.com/paper/2405.09783/) (2024)
- [AutoML-GPT: Automatic Machine Learning with GPT](https://mlsanity.com/paper/2305.02499/) (2023)
- [Can Language Models Solve Olympiad Programming?](https://mlsanity.com/paper/2404.10952/) (2024)
- [MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Tasks](https://mlsanity.com/paper/2304.14979/) (2023)
- [Towards A Unified Agent with Foundation Models](https://mlsanity.com/paper/2307.09668/) (2023)
- [Learning to Learn Faster from Human Feedback with Language Model Predictive Control](https://mlsanity.com/paper/2402.11450/) (2024)
- [Automated Scientific Discovery: From Equation Discovery to Autonomous Discovery Systems](https://mlsanity.com/paper/2305.02251/) (2023)
- [Towards Generalist Robots: A Promising Paradigm via Generative Simulation](https://mlsanity.com/paper/2305.10455/) (2023)

## Built on by (most cited first, 1 of 1)

- [Open-ended Multi-agent Autocurricula via Visual Inspection of Policies with Multi-modal LLMs](https://mlsanity.com/paper/2607.08193/) (2026)

## Abstract (arXiv)

We propose an agent architecture that automates parts of the common reinforcement learning experiment workflow, to enable automated mastery of control domains for embodied agents. To do so, it leverages a VLM to perform some of the capabilities normally required of a human experimenter, including the monitoring and analysis of experiment progress, the proposition of new tasks based on past successes and failures of the agent, decomposing tasks into a sequence of subtasks (skills), and retrieval of the skill to execute - enabling our system to build automated curricula for learning. We believe this is one of the first proposals for a system that leverages a VLM throughout the full experiment cycle of reinforcement learning. We provide a first prototype of this system, and examine the feasibility of current models and techniques for the desired level of automation. For this, we use a standard Gemini model, without additional fine-tuning, to provide a curriculum of skills to a language-conditioned Actor-Critic algorithm, in order to steer data collection so as to aid learning new skills. Data collected in this way is shown to be useful for learning and iteratively improving control policies in a robotics domain. Additional examination of the ability of the system to build a growing library of skills, and to judge the progress of the training of those skills, also shows promising results, suggesting that the proposed architecture provides a potential recipe for fully automated mastery of tasks and domains for embodied agents.