mlsanityData to 7 Oct 2026

Language Reward Modulation for Pretraining Reinforcement Learning

arXiv 2308.12270 ↗First posted 23 Aug 2023cs.LG cs.AIcited by 42

Not on the map. Why?

Abstract (arXiv)

Using learned reward functions (LRFs) as a means to solve sparse-reward reinforcement learning (RL) tasks has yielded some steady progress in task-complexity through the years. In this work, we question whether today's LRFs are best-suited as a direct replacement for task rewards. Instead, we propose leveraging the capabilities of LRFs as a pretraining signal for RL. Concretely, we propose LAnguage Reward Modulated Pretraining (LAMP) which leverages the zero-shot capabilities of Vision-Language Models (VLMs) as a textitpretraining utility for RL as opposed to a downstream task reward. LAMP uses a frozen, pretrained VLM to scalably generate noisy, albeit shaped exploration rewards by computing the contrastive alignment between a highly diverse collection of language instructions and the image observations of an agent in its pretraining environment. LAMP optimizes these rewards in conjunction with standard novelty-seeking exploration rewards with reinforcement learning to acquire a language-conditioned, pretrained policy. Our VLM pretraining approach, which is a departure from previous attempts to use LRFs, can warmstart sample-efficient learning on robot manipulation tasks in RLBench.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 35 of its 51 references that are in the map. First posted on arXiv 23 Aug 2023.

Builds on

35 of 51 references.

References · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  5. 2016Concrete Problems in AI Safetycited by 1,582
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  8. 2018Exploration by Random Network Distillationcited by 966
  9. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  10. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  11. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  12. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
23 more
  1. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  2. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  3. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  4. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  5. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  6. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  7. 2016Variational Intrinsic Controlcited by 262
  8. 2022Masked Visual Pre-training for Motor Controlcited by 246
  9. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  10. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  11. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  12. 2022Masked World Models for Visual Controlcited by 169
  13. 2023Vision-Language Models as Success Detectorscited by 138
  14. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122
  15. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 120
  16. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  17. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 101
  18. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  19. 2022Adversarial Motion Priors Make Good Substitutes for Complex Reward Functionscited by 68
  20. 2016Unsupervised Perceptual Rewards for Imitation Learningcited by 64
  21. 2023Controllability-Aware Unsupervised Skill Discoverycited by 49
  22. 2013Credal Networks under Maximum Entropycited by 13
  23. 2022Skill-Based Reinforcement Learning with Intrinsic Reward Matchingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 20; 2025: 10; 2026: 7.

42 papers.

Sort:

By area:

  1. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  2. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  3. 2025Improving Vision-Language-Action Model with Online Reinforcement Learningcited by 107
  4. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  5. 2024Prediction with Action: Visual Policy Learning via Joint Denoising Processcited by 64
  6. 2023Diffusion Reward: Learning Rewards via Conditional Video Diffusioncited by 56
  7. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  8. 2024VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Drivingcited by 49
  9. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  10. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26
  11. 2024RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agentscited by 20
  12. 2024Learning Reward for Robot Skills Using Large Language Models via Self-Alignmentcited by 19
30 more
  1. 2023LiFT: Unsupervised Reinforcement Learning with Foundation Models as Teacherscited by 14
  2. 2024Scaling Robot Policy Learning via Zero-Shot Labeling with Foundation Modelscited by 14
  3. 2024Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedbackcited by 10
  4. 2023Language-Conditioned Path Planningcited by 9
  5. 2024Language Guided Exploration for RL Agents in Text Environmentscited by 9
  6. 2025SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Modelscited by 9
  7. 2025TeViR: Text-to-Video Reward with Diffusion Models for Efficient Reinforcement Learningcited by 7
  8. 2025Exploratory Retrieval-Augmented Planning For Continual Embodied Instruction Followingcited by 7
  9. 2024Multi-granularity Knowledge Transfer for Continual Reinforcement Learningcited by 6
  10. 2024KOI: Accelerating Online Imitation Learning via Hybrid Key-state Guidancecited by 6
  11. 2025Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learningcited by 6
  12. 2023Learning Reward for Physical Skills using Large Language Modelcited by 5
  13. 2024Redundancy-aware Action Spaces for Robot Learningcited by 5
  14. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 5
  15. 2024CLIP-RLDrive: Human-Aligned Autonomous Driving via CLIP-Based Reward Shaping in Reinforcement Learningcited by 5
  16. 2025World Model Implanting for Test-time Adaptation of Embodied Agentscited by 5
  17. 2024Revision Matters: Generative Design Guided by Revision Editscited by 4
  18. 2024Large Language Model-Enhanced Reinforcement Learning for Generic Bus Holding Control Strategiescited by 4
  19. 2025LaGEA: Language Guided Embodied Agents for Robotic Manipulationcited by 2
  20. 2026MARVL: Multi-Stage Guidance for Robotic Manipulation via Vision-Language Modelscited by 2
  21. 2026LLM-Guided Communication for Cooperative Multi-Agent Reinforcement Learningcited by 2
  22. 2025Vision-Based Generic Potential Function for Policy Alignment in Multi-Agent Reinforcement Learningcited by 1
  23. 2025Remote Sensing Image Intelligent Interpretation with the Language-Centered Perspective: Principles, Methods and Challengescited by 1
  24. 2024From Goal-Conditioned to Language-Conditioned Agents via Vision-Language Modelscited by 0
  25. 2024Fostering Intrinsic Motivation in Reinforcement Learning with Pretrained Foundation Modelscited by 0
  26. 2025FOSSIL: Harnessing Feedback on Suboptimal Samples for Data-Efficient Generalisation with Imitation Learning for Embodied Vision-and-Language Taskscited by 0
  27. 2026MIRA: Memory-Integrated Reinforcement Learning Agent with Limited LLM Guidancecited by 0
  28. 2026Reward-Zero: Language Embedding Driven Implicit Reward Mechanisms for Reinforcement Learningcited by 0
  29. 2026CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learningcited by 0
  30. 2026Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skillscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
  2. 2024Code as Reward: Empowering Reinforcement Learning with VLMs
  3. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning
  4. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback
  5. 2026From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Models
  6. 2023Vision-Language Models as a Source of Rewards
  7. 2025Revisiting the Learning Objectives of Vision-Language Reward Models
  8. 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning