mlsanityData to 14 Oct 2026

Reinforcement Learning Friendly Vision-Language Model for Minecraft

arXiv 2303.10571 ↗First posted 19 Mar 2023cs.LG cs.CVcited by 14

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

One of the essential missions in the AI research community is to build an autonomous embodied agent that can achieve high-level performance across a wide spectrum of tasks. However, acquiring or manually designing rewards for all open-ended tasks is unrealistic. In this paper, we propose a novel cross-modal contrastive learning framework architecture, CLIP4MC, aiming to learn a reinforcement learning (RL) friendly vision-language model (VLM) that serves as an intrinsic reward function for open-ended tasks. Simply utilizing the similarity between the video snippet and the language prompt is not RL-friendly since standard VLMs may only capture the similarity at a coarse level. To achieve RL-friendliness, we incorporate the task completion degree into the VLM training objective, as this information can assist agents in distinguishing the importance between different states. Moreover, we provide neat YouTube datasets based on the large-scale YouTube database provided by MineDojo. Specifically, two rounds of filtering operations guarantee that the dataset covers enough essential information and that the video-text pair is highly correlated. Empirically, we demonstrate that the proposed method achieves better performance on RL tasks compared with baselines. The code and datasets are available at https://github.com/PKU-RL/CLIP4MC.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 41 references that are in the map. First posted on arXiv 19 Mar 2023.

Builds on

24 of 41 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  7. 2023Mastering Diverse Domains through World Modelscited by 1,253
  8. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  9. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  10. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  11. 2021Extract Free Dense Labels from CLIPcited by 583
  12. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
12 more
  1. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  2. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 215
  3. 2016A Deep Hierarchical Approach to Lifelong Learning in Minecraftcited by 90
  4. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88
  5. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
  6. 2021MDMMT: Multidomain Multimodal Transformer for Video Retrievalcited by 78
  7. 2017Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learningcited by 77
  8. 2023Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Predictioncited by 39
  9. 2021JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learningcited by 28
  10. 2019The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priorscited by 26
  11. 2022MineRL Diamond 2021 Competition: Overview, Results, and Lessons Learnedcited by 26
  12. 2022Retrospective on the 2021 BASALT Competition on Learning from Human Feedbackcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 2; 2025: 5; 2026: 1.

14 papers.

Sort:

By area:

  1. 2023MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perceptioncited by 67
  2. 2023Vision-Language Models as a Source of Rewardscited by 66
  3. 2024Code as Reward: Empowering Reinforcement Learning with VLMscited by 39
  4. 2023Skill Reinforcement Learning and Planning for Open-World Long-Horizon Taskscited by 36
  5. 2023MCU: An Evaluation Framework for Open-Ended Game Agentscited by 34
  6. 2023Creative Agents: Empowering Agents with Imagination for Creative Taskscited by 24
  7. 2025Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Modelscited by 15
  8. 2023Learning from Visual Observation via Offline Pretrained State-to-Go Transformercited by 14
  9. 2025Position: Interactive Generative Video as Next-Generation Game Enginecited by 13
  10. 2024MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agentscited by 11
  11. 2025Open-World Skill Discovery from Unsegmented Demonstrationscited by 7
  12. 2025GuessBench: Sensemaking Multimodal Creativity in the Wildcited by 1
2 more
  1. 2025VITA: Zero-Shot Value Functions via Test-Time Adaptation of Vision-Language Modelscited by 1
  2. 2026VLM-AR3L: Vision-Language Models for Absolute and Relative Rewards in Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learning
  2. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback
  3. 2025JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mouse
  4. 2026Training a Conditioned Video Game Agent on a VLM Annotated Dataset
  5. 2024ViSTa Dataset: Do vision-language models understand sequential tasks?
  6. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning
  7. 2025Enhancing Vision-Language Model Training with Reinforcement Learning in Synthetic Worlds for Real-World Success
  8. 2025VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators