# GoalLadder: Incremental Goal Discovery with Vision-Language Models

arXiv: https://arxiv.org/abs/2506.16396 · HTML: https://mlsanity.com/paper/2506.16396/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 32 of its 41 references that are in the map.
- First posted on arXiv 19 Jun 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [Language to Rewards for Robotic Skill Synthesis](https://mlsanity.com/paper/2306.08647/) (2023)
- [PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training](https://mlsanity.com/paper/2106.05091/) (2021)
- [Reward Design with Language Models](https://mlsanity.com/paper/2303.00001/) (2023)
- [What's "up" with vision-language models? Investigating their struggle with spatial reasoning](https://mlsanity.com/paper/2310.19785/) (2023)
- [An Introduction to Vision-Language Modeling](https://mlsanity.com/paper/2405.17247/) (2024)
- [Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning](https://mlsanity.com/paper/2310.12921/) (2023)
- [Text2Reward: Reward Shaping with Language Models for Reinforcement Learning](https://mlsanity.com/paper/2309.11489/) (2023)
- [RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback](https://mlsanity.com/paper/2402.03681/) (2024)
- [RoboCLIP: One Demonstration is Enough to Learn Robot Policies](https://mlsanity.com/paper/2310.07899/) (2023)
- [Motif: Intrinsic Motivation from Artificial Intelligence Feedback](https://mlsanity.com/paper/2310.00166/) (2023)
- [Vision-Language Models as a Source of Rewards](https://mlsanity.com/paper/2312.09187/) (2023)
- [Vision-Language Models Provide Promptable Representations for Reinforcement Learning](https://mlsanity.com/paper/2402.02651/) (2024)
- [Can Language Models Act as Knowledge Bases at Scale?](https://mlsanity.com/paper/2402.14273/) (2024)
- [Trajectory Improvement and Reward Learning from Comparative Language Feedback](https://mlsanity.com/paper/2410.06401/) (2024)
- [LiFT: Unsupervised Reinforcement Learning with Foundation Models as Teachers](https://mlsanity.com/paper/2312.08958/) (2023)
- [Variational Predictive Routing with Nested Subjective Timescales](https://mlsanity.com/paper/2110.11236/) (2021)
- [S3D: A Simple and Cost-Effective Self-Speculative Decoding Scheme for Low-Memory GPUs](https://mlsanity.com/paper/2405.20314/) (2024)
- [Long-horizon video prediction using a dynamic latent hierarchy](https://mlsanity.com/paper/2212.14376/) (2022)

## Built on by (most cited first, 3 of 3)

- [Reinforcement Learning for Large Model: A Survey](https://mlsanity.com/paper/2508.08189/) (2025)
- [SpecRLBench: A Benchmark for Generalization in Specification-Guided Reinforcement Learning](https://mlsanity.com/paper/2604.24729/) (2026)
- [Intrinsic Robot Rewarding: Reusing VLA Representations for Autonomous Evaluation and Policy Improvement](https://mlsanity.com/paper/2609.17115/) (2026)

## Abstract (arXiv)

Natural language can offer a concise and human-interpretable means of specifying reinforcement learning (RL) tasks. The ability to extract rewards from a language instruction can enable the development of robotic systems that can learn from human guidance; however, it remains a challenging problem, especially in visual environments. Existing approaches that employ large, pretrained language models either rely on non-visual environment representations, require prohibitively large amounts of feedback, or generate noisy, ill-shaped reward functions. In this paper, we propose a novel method, GoalLadder, that leverages vision-language models (VLMs) to train RL agents from a single language instruction in visual environments. GoalLadder works by incrementally discovering states that bring the agent closer to completing a task specified in natural language. To do so, it queries a VLM to identify states that represent an improvement in agent's task progress and to rank them using pairwise comparisons. Unlike prior work, GoalLadder does not trust VLM's feedback completely; instead, it uses it to rank potential goal states using an ELO-based rating system, thus reducing the detrimental effects of noisy VLM feedback. Over the course of training, the agent is tasked with minimising the distance to the top-ranked goal in a learned embedding space, which is trained on unlabelled visual data. This key feature allows us to bypass the need for abundant and accurate feedback typically required to train a well-shaped reward function. We demonstrate that GoalLadder outperforms existing related methods on classic control and robotic manipulation environments with the average final success rate of ~95% compared to only ~45% of the best competitor.