mlsanityData to 7 Oct 2026

Fostering Intrinsic Motivation in Reinforcement Learning with Pretrained Foundation Models

arXiv 2410.07404 ↗First posted 9 Oct 2024cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Exploration remains a significant challenge in reinforcement learning, especially in environments where extrinsic rewards are sparse or non-existent. The recent rise of foundation models, such as CLIP, offers an opportunity to leverage pretrained, semantically rich embeddings that encapsulate broad and reusable knowledge. In this work we explore the potential of these foundation models not just to drive exploration, but also to analyze the critical role of the episodic novelty term in enhancing exploration effectiveness of the agent. We also investigate whether providing the intrinsic module with complete state information -- rather than just partial observations -- can improve exploration, despite the difficulties in handling small variations within large state spaces. Our experiments in the MiniGrid domain reveal that intrinsic modules can effectively utilize full state information, significantly increasing sample efficiency while learning an optimal policy. Moreover, we show that the embeddings provided by foundation models are sometimes even better than those constructed by the agent during training, further accelerating the learning process, especially when coupled with the episodic novelty term to enhance exploration.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 27 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

23 of 27 references.

References · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  5. 2018Exploration by Random Network Distillationcited by 966
  6. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  7. 2017Noisy Networks for Explorationcited by 262
  8. 2017Count-Based Exploration with Neural Density Modelscited by 255
  9. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  10. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  11. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  12. 2018Episodic Curiosity through Reachabilitycited by 109
11 more
  1. 2020Learning with AMIGo: Adversarially Motivated Intrinsic Goalscited by 86
  2. 2023Vision-Language Models as a Source of Rewardscited by 65
  3. 2021Imitation Learning: Progress, Taxonomies and Challengescited by 56
  4. 2021Adversarially Guided Actor-Criticcited by 47
  5. 2023Language Reward Modulation for Pretraining Reinforcement Learningcited by 42
  6. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26
  7. 2023Accelerating Reinforcement Learning of Robotic Manipulations via Feedback from Large Language Modelscited by 22
  8. 2023A Study of Global and Episodic Bonuses for Exploration in Contextual MDPscited by 19
  9. 2022LECO: Learnable Episodic Count for Task-Specific Intrinsic Rewardcited by 11
  10. 2022Foundation Models for Semantic Novelty in Reinforcement Learningcited by 9
  11. 2022An Evaluation Study of Intrinsic Motivation Techniques applied to Reinforcement Learning over Hard Exploration Environmentscited by 8

Built on by

Not cited yet.