Data to 7 Oct 2026

Control of Memory, Active Perception, and Action in Minecraft

arXiv 1605.09128 ↗First posted 30 May 2016cs.AI cs.CV cs.LGcited by 61

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce a new set of reinforcement learning (RL) tasks in Minecraft (a flexible 3D world). We then use these tasks to systematically compare and contrast existing deep reinforcement learning (DRL) architectures with our new memory-based DRL architectures. These tasks are designed to emphasize, in a controllable manner, issues that pose challenges for RL methods including partial observability (due to first-person visual observations), delayed rewards, high-dimensional visual observations, and the need to use active perception in a correct manner so as to perform well in the tasks. While these tasks are conceptually simple to describe, by virtue of having all of these challenges simultaneously they are difficult for current DRL architectures. Additionally, we evaluate the generalization performance of the architectures on environments not used during training. The experimental results show that our new architectures generalize to unseen environments better than existing DRL architectures.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 May 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 1; 2018: 0; 2019: 2; 2020: 0; 2021: 23; 2022: 17; 2023: 3; 2024: 9; 2025: 5; 2026: 1.

61 papers.

Sort:

By area:

2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
2022Memory-Based Model Editing at Scalecited by 443
2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
2021Structured Scene Memory for Vision-Language Navigationcited by 95
2021AndroidEnv: A Reinforcement Learning Platform for Androidcited by 90
2019Deep Reinforcement Learning from Policy-Dependent Human Feedbackcited by 56
2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 50
2021Scene-Intuitive Agent for Remote Embodied Visual Groundingcited by 49
2022Local-Global Context Aware Transformer for Language-Guided Video Segmentationcited by 41
2021The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priorscited by 40
2024Online Adaptation of Language Models with a Memory of Amortized Contextscited by 35

38 more

2025Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learningcited by 34
2023MCU: An Evaluation Framework for Open-Ended Game Agentscited by 33
2025ReflAct: World-Grounded Decision Making in LLM Agents via Goal-State Reflectioncited by 30
2021JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learningcited by 28
2021Synthetic Returns for Long-Term Credit Assignmentcited by 27
2022Large-Scale Retrieval for Reinforcement Learningcited by 24
2022Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learningcited by 22
2022Object Memory Transformer for Object Goal Navigationcited by 19
2021A Survey on Deep Reinforcement Learning for Audio-Based Applicationscited by 17
2021Teaching Agents how to Map: Spatial Reasoning for Multi-Object Navigationcited by 17
2021Learning Reward Machines: A Study in Partially Observable Reinforcement Learningcited by 16
2022Navigating to Objects in Unseen Environments by Distance Predictioncited by 15
2022Light-weight probing of unsupervised representations for Reinforcement Learningcited by 15
2022Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learningcited by 12
2024LEGENT: Open Platform for Embodied Agentscited by 12
2021MaAST: Map Attention with Semantic Transformersfor Efficient Visual Navigationcited by 11
2022Explain My Surprise: Learning Efficient Long-Term Memory by Predicting Uncertain Outcomescited by 11
2024ActiView: Evaluating Active Perception Ability for Multimodal Large Language Modelscited by 11
2017Explore, Exploit or Listen: Combining Human Feedback and Policy Model to Speed up Deep Reinforcement Learning in 3D Worldscited by 8
2021Modeling the Interaction between Agents in Cooperative Multi-Agent Reinforcement Learningcited by 7
2024RICE: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanationcited by 7
2021Reinforcement Learning-Based Coverage Path Planning with Implicit Cellular Decompositioncited by 6
2022Multi-Environment Pretraining Enables Transfer to Action Limited Datasetscited by 5
2024Uncertainty Representations in State-Space Layers for Deep Reinforcement Learning under Partial Observabilitycited by 5
2024Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluationcited by 5
2022Collaborative Training of Heterogeneous Reinforcement Learning Agents in Environments with Sparse Rewards: What and When to Share?cited by 4
2025PuzzleJAX: A Benchmark for Reasoning and Learningcited by 4
2021SGoLAM: Simultaneous Goal Localization and Mapping for Multi-Object Goal Navigationcited by 3
2021Off-Dynamics Inverse Reinforcement Learning from Hetero-Domaincited by 3
2024Deep Reinforcement Learning Based Toolpath Generation for Thermal Uniformity in Laser Powder Bed Fusion Processcited by 3
2024Online Control-Informed Learningcited by 3
2021Towards robust and domain agnostic reinforcement learning competitionscited by 2
2022Influence-Augmented Local Simulators: A Scalable Solution for Fast Deep RL in Large Networked Systemscited by 2
2025pix2pockets: Shot Suggestions in 8-Ball Pool from a Single Image in the Wildcited by 2
2021Learning to Reach, Swim, Walk and Fly in One Trial: Data-Driven Control with Scarce Data and Side Informationcited by 1
2021Graph Convolutional Memory using Topological Priorscited by 1
2023Deep Generative Models for Decision-Making and Controlcited by 1
2024CycLight: learning traffic signal cooperation with a cycle-level strategycited by 1

50 most cited of 61 All 61 (JSON)