mlsanityData to 7 Oct 2026

VLM-Guided Experience Replay

arXiv 2602.01915 ↗First posted 2 Feb 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Recent advances in Large Language Models (LLMs) and Vision-Language Models (VLMs) have enabled powerful semantic and multimodal reasoning capabilities, creating new opportunities to enhance sample efficiency, high-level planning, and interpretability in reinforcement learning (RL). While prior work has integrated LLMs and VLMs into various components of RL, the replay buffer, a core component for storing and reusing experiences, remains unexplored. We propose addressing this gap by leveraging VLMs to guide the prioritization of experiences in the replay buffer. Our key idea is to use a frozen, pre-trained VLM as an automated evaluator to identify and prioritize promising sub-trajectories from the agent's experiences. Across scenarios, including game-playing and robotics, spanning both discrete and continuous domains, agents trained with our proposed prioritization method achieve 15-57% higher average success rates and improve sample efficiency by 35-55% compared to previous approaches. Project page: https://esharony.me/projects/vlm-rb/

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 52 of its 64 references that are in the map. First posted on arXiv 2 Feb 2026.

Builds on

52 of 64 references.

References · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  10. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  11. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  12. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
38 more
  1. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  2. 2015Prioritized Experience Replaycited by 1,070
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  4. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  5. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  6. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  7. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  8. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  9. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  10. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  11. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  12. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
  13. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 153
  14. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  15. 2019A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithmscited by 136
  16. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  17. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  18. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 111
  19. 2025PerceptionLM: Open-Access Data and Models for Detailed Visual Understandingcited by 88
  20. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  21. 2023TorchRL: A data-driven decision-making library for PyTorchcited by 59
  22. 2019Experience Replay Optimizationcited by 47
  23. 2018Remember and Forget for Experience Replaycited by 42
  24. 2020Experience Replay with Likelihood-free Importance Weightscited by 42
  25. 2018Investigating Human Priors for Playing Video Gamescited by 39
  26. 2019Curiosity-Driven Experience Prioritization via Density Estimationcited by 27
  27. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26
  28. 2021Regret Minimization Experience Replay in Off-Policy Reinforcement Learningcited by 25
  29. 2022Prioritizing Samples in Reinforcement Learning with Reducible Losscited by 20
  30. 2024ExploRLLM: Guiding Exploration in Reinforcement Learning with Large Language Modelscited by 19
  31. 2022Topological Experience Replaycited by 17
  32. 2025Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Modelscited by 15
  33. 2021Large Batch Experience Replaycited by 14
  34. 2025The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learningcited by 10
  35. 2022Foundation Models for Semantic Novelty in Reinforcement Learningcited by 9
  36. 2025VARP: Reinforcement Learning from Vision-Language Model Feedback with Agent Regularized Preferencescited by 8
  37. 2021Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environmentcited by 7
  38. 2025Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learningcited by 6

50 most cited of 52 All 52 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning
  2. 2025Improving RL Exploration for LLM Reasoning through Retrospective Replay
  3. 2025RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning
  4. 2026GameVerse: Can Vision-Language Models Learn from Video-based Reflection?
  5. 2024Code as Reward: Empowering Reinforcement Learning with VLMs
  6. 2026VLMs for Videogame Data Annotation
  7. 2025VLM Q-Learning: Aligning Vision-Language Models for Interactive Decision-Making
  8. 2024Augmenting Replay in World Models for Continual Reinforcement Learning