mlsanityData to 7 Oct 2026

STEVE-Audio: Expanding the Goal Conditioning Modalities of Embodied Agents in Minecraft

arXiv 2412.00949 ↗First posted 1 Dec 2024cs.LG cs.AI cs.ROcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Recently, the STEVE-1 approach has been introduced as a method for training generative agents to follow instructions in the form of latent CLIP embeddings. In this work, we present a methodology to extend the control modalities by learning a mapping from new input modalities to the latent goal space of the agent. We apply our approach to the challenging Minecraft domain, and extend the goal conditioning to include the audio modality. The resulting audio-conditioned agent is able to perform on a comparable level to the original text-conditioned and visual-conditioned agents. Specifically, we create an Audio-Video CLIP foundation model for Minecraft and an audio prior network which together map audio samples to the latent goal space of the STEVE-1 policy. Additionally, we highlight the tradeoffs that occur when conditioning on different modalities. Our training code, evaluation code, and Audio-Video CLIP foundation model for Minecraft are made open-source to help foster further research into multi-modal generalist sequential decision-making agents.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 30 of its 34 references that are in the map. First posted on arXiv 1 Dec 2024.

Builds on

30 of 34 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models10 Nov
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
  3. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge17 Jun

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  6. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  7. 2021AST: Audio Spectrogram Transformercited by 634
  8. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  9. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  10. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  11. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  12. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
15 more
  1. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  2. 2023MUTEX: Learning Unified Policies from Multimodal Task Specificationscited by 70
  3. 2023MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perceptioncited by 67
  4. 2023Vision-Language Models as a Source of Rewardscited by 65
  5. 2024MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Controlcited by 52
  6. 2023GROOT: Learning to Follow Instructions by Watching Gameplay Videoscited by 42
  7. 2022Face Generation and Editing with StyleGAN: A Surveycited by 27
  8. 2022AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environmentscited by 26
  9. 2023Accommodating Audio Modality in CLIP for Multimodal Processingcited by 18
  10. 2023Towards Solving Fuzzy Tasks with Human Feedback: A Retrospective of the MineRL BASALT 2022 Competitioncited by 15
  11. 2023Contrastive Language, Action, and State Pre-training for Robot Learningcited by 15
  12. 2024Zero-shot Imitation Policy via Search in Demonstration Datasetcited by 13
  13. 2021Critic Guided Segmentation of Rewarding Objects in First-Person Viewscited by 12
  14. 2022Behavioral Cloning via Search in Video PreTraining Latent Spacecited by 9
  15. 2023CLIP-Motion: Learning Reward Functions for Robotic Actions Using Consecutive Observationscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policy
  2. 2024STEVE Series: Step-by-Step Construction of Agent Systems in Minecraft
  3. 2024TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraft
  4. 2023See and Think: Embodied Agent in Virtual Environment
  5. 2025OpenHA: A Series of Open-Source Hierarchical Agentic Models in Minecraft
  6. 2025CrafterDojo: A Suite of Foundation Models for Building Open-Ended Embodied Agents in Crafter
  7. 2025Training Agents Inside of Scalable World Models
  8. 2024OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents