# Learning Instruction-Following Policies through Open-Ended Instruction Relabeling with Large Language Models

arXiv: https://arxiv.org/abs/2506.20061 · HTML: https://mlsanity.com/paper/2506.20061/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 24 Jun 2025.

## Builds on (most cited first, 20 of 20)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [Visual Reinforcement Learning with Imagined Goals](https://mlsanity.com/paper/1807.04742/) (2018)
- [Prioritized Level Replay](https://mlsanity.com/paper/2010.03934/) (2020)
- [Guiding Pretraining in Reinforcement Learning with Large Language Models](https://mlsanity.com/paper/2302.06692/) (2023)
- [Goal-Conditioned Reinforcement Learning: Problems and Solutions](https://mlsanity.com/paper/2201.08299/) (2022)
- [A Survey of Reinforcement Learning Informed by Natural Language](https://mlsanity.com/paper/1906.03926/) (2019)
- [Unsupervised Control Through Non-Parametric Discriminative Rewards](https://mlsanity.com/paper/1811.11359/) (2018)
- [Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning](https://mlsanity.com/paper/2007.02832/) (2020)
- [Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning](https://mlsanity.com/paper/2402.16801/) (2024)
- [Simplifying Deep Temporal Difference Learning](https://mlsanity.com/paper/2407.04811/) (2024)
- [The Wisdom of Hindsight Makes Language Models Better Instruction Followers](https://mlsanity.com/paper/2302.05206/) (2023)
- [Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text](https://mlsanity.com/paper/2005.09382/) (2020)
- [Exploration via Hindsight Goal Generation](https://mlsanity.com/paper/1906.04279/) (2019)
- [Inferring Rewards from Language in Context](https://mlsanity.com/paper/2204.02515/) (2022)
- [M3HF: Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality](https://mlsanity.com/paper/2503.02077/) (2025)

## Built on by (most cited first, 2 of 2)

- [Recontextualization Mitigates Specification Gaming without Modifying the Specification](https://mlsanity.com/paper/2512.19027/) (2025)
- [Learning More from Less: Reinforcement Learning from Hindsight](https://mlsanity.com/paper/2607.09042/) (2026)

## Abstract (arXiv)

Developing effective instruction-following policies in reinforcement learning remains challenging due to the reliance on extensive human-labeled instruction datasets and the difficulty of learning from sparse rewards. In this paper, we propose a novel approach that leverages the capabilities of large language models (LLMs) to automatically generate open-ended instructions retrospectively from previously collected agent trajectories. Our core idea is to employ LLMs to relabel unsuccessful trajectories by identifying meaningful subtasks the agent has implicitly accomplished, thereby enriching the agent's training data and substantially alleviating reliance on human annotations. Through this open-ended instruction relabeling, we efficiently learn a unified instruction-following policy capable of handling diverse tasks within a single policy. We empirically evaluate our proposed method in the challenging Craftax environment, demonstrating clear improvements in sample efficiency, instruction coverage, and overall policy performance compared to state-of-the-art baselines. Our results highlight the effectiveness of utilizing LLM-guided open-ended instruction relabeling to enhance instruction-following reinforcement learning.