mlsanityData to 14 Oct 2026

LiFT: Unsupervised Reinforcement Learning with Foundation Models as Teachers

arXiv 2312.08958 ↗First posted 14 Dec 2023cs.LG cs.AI cs.ROcited by 14

Not on the map. Why?

Abstract (arXiv)

We propose a framework that leverages foundation models as teachers, guiding a reinforcement learning agent to acquire semantically meaningful behavior without human feedback. In our framework, the agent receives task instructions grounded in a training environment from large language models. Then, a vision-language model guides the agent in learning the multi-task language-conditioned policy by providing reward feedback. We demonstrate that our method can learn semantically meaningful skills in a challenging open-ended MineDojo environment while prior unsupervised skill discovery methods struggle. Additionally, we discuss observed challenges of using off-the-shelf foundation models as teachers and our efforts to address them.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 44 of its 47 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

44 of 47 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  6. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  8. 2018Exploration by Random Network Distillationcited by 993
  9. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  10. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  11. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  12. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
32 more
  1. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  2. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  3. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  4. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  5. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  6. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  7. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 250
  8. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 244
  9. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  10. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 215
  11. 2020The NetHack Learning Environmentcited by 198
  12. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 187
  13. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 182
  14. 2021Reinforcement Learning with Prototypical Representationscited by 179
  15. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
  16. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 134
  17. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 133
  18. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 133
  19. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
  20. 2021APS: Active Pretraining with Successor Featurescited by 112
  21. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 97
  22. 2023Compositional Foundation Models for Hierarchical Planningcited by 95
  23. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88
  24. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 86
  25. 2021Demonstration-Guided Reinforcement Learning with Learned Skillscited by 76
  26. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  27. 2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 70
  28. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 68
  29. 2022Lipschitz-constrained Unsupervised Skill Discoverycited by 62
  30. 2023Language Reward Modulation for Pretraining Reinforcement Learningcited by 43
  31. 2023SPRINT: Scalable Policy Pre-Training via Language Instruction Relabelingcited by 24
  32. 2023Language-Conditioned Path Planningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 7; 2026: 4.

14 papers.

Sort:

By area:

  1. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 156
  2. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 148
  3. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 60
  4. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26
  5. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  6. 2025Automated Skill Discovery for Language Agents through Exploration and Iterative Feedbackcited by 12
  7. 2025Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Makingcited by 5
  8. 2025VLA Model Post-Training via Action-Chunked PPO and Self Behavior Cloningcited by 4
  9. 2025GoalLadder: Incremental Goal Discovery with Vision-Language Modelscited by 3
  10. 2026Multi-Modal Data-Enhanced Foundation Models for Prediction and Control in Wireless Networks: A Surveycited by 3
  11. 2025PRISM: Projection-based Reward Integration for Scene-Aware Real-to-Sim-to-Real Transfer with Few Demonstrationscited by 2
  12. 2025LaGEA: Language Guided Embodied Agents for Robotic Manipulationcited by 2
2 more
  1. 2026MARVL: Multi-Stage Guidance for Robotic Manipulation via Vision-Language Modelscited by 2
  2. 2026VLM-AR3L: Vision-Language Models for Absolute and Relative Rewards in Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
  2. 2023Towards A Unified Agent with Foundation Models
  3. 2024Words as Beacons: Guiding RL Agents with High-Level Language Prompts
  4. 2026Leveraging Human Feedback for Semantically-Relevant Skill Discovery
  5. 2026SKILLER: Language-Level Reinforcement Learning for Reusable Skill Extraction in Small Language Models
  6. 2024EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data
  7. 2025InnateCoder: Learning Programmatic Options with Foundation Models
  8. 2025SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models