mlsanityData to 14 Oct 2026

FoMo Rewards: Can we cast foundation models as reward functions?

arXiv 2312.03881 ↗First posted 6 Dec 2023cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

We explore the viability of casting foundation models as generic reward functions for reinforcement learning. To this end, we propose a simple pipeline that interfaces an off-the-shelf vision model with a large language model. Specifically, given a trajectory of observations, we infer the likelihood of an instruction describing the task that the user wants an agent to perform. We show that this generic likelihood function exhibits the characteristics ideally expected from a reward function: it associates high values with the desired behaviour and lower values for several similar, but incorrect policies. Overall, our work opens the possibility of designing open-ended agents for interactive tasks via foundation models.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 141 of its 159 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

141 of 159 references.

References · 141

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
38 more
  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2015Continuous control with deep reinforcement learningcited by 3,407
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  7. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  11. 2022Emergent Abilities of Large Language Modelscited by 2,512
  12. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  13. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  14. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  15. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  16. 2019Language Models as Knowledge Bases?cited by 1,936
  17. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  18. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  19. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
  20. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  21. 2023Mastering Diverse Domains through World Modelscited by 1,253
  22. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  23. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  24. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  25. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  26. 2020Mastering Atari with Discrete World Modelscited by 1,007
  27. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  28. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  29. 2017Deep Reinforcement Learning that Matterscited by 754
  30. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  31. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  32. 2022Large Language Models Can Self-Improvecited by 664
  33. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  34. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  35. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  36. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  37. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  38. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522

50 most cited of 141 All 141 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 0; 2026: 1.

4 papers.

  1. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 148
  2. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26
  3. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 6
  4. 2026MARVL: Multi-Stage Guidance for Robotic Manipulation via Vision-Language Modelscited by 2