mlsanityData to 7 Oct 2026

Behavioral Cloning via Search in Video PreTraining Latent Space

arXiv 2212.13326 ↗First posted 27 Dec 2022cs.LG cs.AI cs.CVcited by 9

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Our aim is to build autonomous agents that can solve tasks in environments like Minecraft. To do so, we used an imitation learning-based approach. We formulate our control problem as a search problem over a dataset of experts' demonstrations, where the agent copies actions from a similar demonstration trajectory of image-action pairs. We perform a proximity search over the BASALT MineRL-dataset in the latent representation of a Video PreTraining model. The agent copies the actions from the expert trajectory as long as the distance between the state representations of the agent and the selected expert trajectory from the dataset do not diverge. Then the proximity search is repeated. Our approach can effectively recover meaningful demonstration trajectories and show human-like behavior of an agent in the Minecraft environment.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 7 of its 12 references that are in the map. First posted on arXiv 27 Dec 2022.

Builds on

7 of 12 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos23 Jun
  2. 2021Critic Guided Segmentation of Rewarding Objects in First-Person Views20 Jul

Other references · 5

  1. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  2. 2021The MineRL BASALT Competition on Learning from Human Feedbackcited by 24
  3. 2022Planning with RL and episodic-memory behavioral priorscited by 8
  4. 2019Modularization of End-to-End Learning: Case Study in Arcade Gamescited by 6
  5. 2022Solving Learn-to-Race Autonomous Racing Challenge by Planning in Latent Spacecited by 4

Built on by

Cited by year

Citing papers per year: 2023: 7; 2024: 2.

9 papers.

  1. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  2. 2023Towards Solving Fuzzy Tasks with Human Feedback: A Retrospective of the MineRL BASALT 2022 Competitioncited by 15
  3. 2023UniTeam: Open Vocabulary Mobile Manipulation Challengecited by 15
  4. 2023BEDD: The MineRL BASALT Evaluation and Demonstrations Dataset for Training and Benchmarking Agents that Solve Fuzzy Taskscited by 8
  5. 2023Benchmarks for Physical Reasoning AIcited by 8
  6. 2023Language-Conditioned Semantic Search-Based Policy for Robotic Manipulation Taskscited by 5
  7. 2023Shape complexity estimation using VAEcited by 4
  8. 2024Natural Language as Policies: Reasoning for Coordinate-Level Embodied Control with LLMscited by 4
  9. 2024STEVE-Audio: Expanding the Goal Conditioning Modalities of Embodied Agents in Minecraftcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Zero-shot Imitation Policy via Search in Demonstration Dataset
  2. 2023Behavioral Cloning via Search in Embedded Demonstration Dataset
  3. 2025Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policy
  4. 2024Behavioural Cloning in VizDoom
  5. 2020Imitating Unknown Policies via Exploration
  6. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
  7. 2021Combining Learning from Human Feedback and Knowledge Engineering to Solve Hierarchical Tasks in Minecraft
  8. 2025Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations