mlsanityData to 7 Oct 2026

GoalLadder: Incremental Goal Discovery with Vision-Language Models

arXiv 2506.16396 ↗First posted 19 Jun 2025cs.LGcited by 3

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Natural language can offer a concise and human-interpretable means of specifying reinforcement learning (RL) tasks. The ability to extract rewards from a language instruction can enable the development of robotic systems that can learn from human guidance; however, it remains a challenging problem, especially in visual environments. Existing approaches that employ large, pretrained language models either rely on non-visual environment representations, require prohibitively large amounts of feedback, or generate noisy, ill-shaped reward functions. In this paper, we propose a novel method, GoalLadder, that leverages vision-language models (VLMs) to train RL agents from a single language instruction in visual environments. GoalLadder works by incrementally discovering states that bring the agent closer to completing a task specified in natural language. To do so, it queries a VLM to identify states that represent an improvement in agent's task progress and to rank them using pairwise comparisons. Unlike prior work, GoalLadder does not trust VLM's feedback completely; instead, it uses it to rank potential goal states using an ELO-based rating system, thus reducing the detrimental effects of noisy VLM feedback. Over the course of training, the agent is tasked with minimising the distance to the top-ranked goal in a learned embedding space, which is trained on unlabelled visual data. This key feature allows us to bypass the need for abundant and accurate feedback typically required to train a well-shaped reward function. We demonstrate that GoalLadder outperforms existing related methods on classic control and robotic manipulation environments with the average final success rate of ~95% compared to only ~45% of the best competitor.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 41 references that are in the map. First posted on arXiv 19 Jun 2025.

Builds on

32 of 41 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback6 Feb
  2. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learning5 Feb
  3. 2023Vision-Language Models as a Source of Rewards14 Dec
  4. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoning30 Oct
  5. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning19 Oct

Other references · 27

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2013Auto-Encoding Variational Bayescited by 3,980
  7. 2019Language Models as Knowledge Bases?cited by 1,922
  8. 2016OpenAI Gymcited by 1,684
  9. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  10. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  11. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  12. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
15 more
  1. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  2. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  3. 2023Reward Design with Language Modelscited by 257
  4. 2024An Introduction to Vision-Language Modelingcited by 175
  5. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
  6. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  7. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  8. 2024Can Language Models Act as Knowledge Bases at Scale?cited by 17
  9. 2024Trajectory Improvement and Reward Learning from Comparative Language Feedbackcited by 17
  10. 2023LiFT: Unsupervised Reinforcement Learning with Foundation Models as Teacherscited by 14
  11. 2021Variational Predictive Routing with Nested Subjective Timescalescited by 9
  12. 2024S3D: A Simple and Cost-Effective Self-Speculative Decoding Scheme for Low-Memory GPUscited by 6
  13. 2022Long-horizon video prediction using a dynamic latent hierarchycited by 4
  14. 2020Episodic Memory for Learning Subjective-Timescale Modelscited by 3
  15. 2023Language Models as Knowledge Bases for Visual Word Sense Disambiguationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Reinforcement Learning for Large Model: A Surveycited by 8
  2. 2026SpecRLBench: A Benchmark for Generalization in Specification-Guided Reinforcement Learningcited by 1
  3. 2026Intrinsic Robot Rewarding: Reusing VLA Representations for Autonomous Evaluation and Policy Improvementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
  2. 2024From Goal-Conditioned to Language-Conditioned Agents via Vision-Language Models
  3. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learning
  4. 2026VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agents
  5. 2026PRTS: A Primitive Reasoning and Tasking System via Contrastive Representations
  6. 2025Training-free Generation of Temporally Consistent Rewards from VLMs
  7. 2023Towards A Unified Agent with Foundation Models
  8. 2026MARVL: Multi-Stage Guidance for Robotic Manipulation via Vision-Language Models