# Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft

arXiv: https://arxiv.org/abs/2312.09238 · HTML: https://mlsanity.com/paper/2312.09238/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 55 references that are in the map.
- First posted on arXiv 14 Dec 2023.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training](https://mlsanity.com/paper/2210.00030/) (2022)
- [Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos](https://mlsanity.com/paper/2206.11795/) (2022)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks](https://mlsanity.com/paper/2306.13831/) (2023)
- [Language to Rewards for Robotic Skill Synthesis](https://mlsanity.com/paper/2306.08647/) (2023)
- [Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory](https://mlsanity.com/paper/2305.17144/) (2023)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Large-Scale Study of Curiosity-Driven Learning](https://mlsanity.com/paper/1808.04355/) (2018)
- [Human-level performance in first-person multiplayer games with population-based deep reinforcement learning](https://mlsanity.com/paper/1807.01281/) (2018)
- [Reward Design with Language Models](https://mlsanity.com/paper/2303.00001/) (2023)
- [MineRL: A Large-Scale Dataset of Minecraft Demonstrations](https://mlsanity.com/paper/1907.13440/) (2019)
- [LIV: Language-Image Representations and Rewards for Robotic Control](https://mlsanity.com/paper/2306.00958/) (2023)
- [Guiding Pretraining in Reinforcement Learning with Large Language Models](https://mlsanity.com/paper/2302.06692/) (2023)
- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)

## Built on by (most cited first, 30 of 61)

- [Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods](https://mlsanity.com/paper/2404.00282/) (2024)
- [A Survey on Large Language Model-Based Game Agents](https://mlsanity.com/paper/2404.02039/) (2024)
- [A Systematic Survey on Large Language Models for Algorithm Design](https://mlsanity.com/paper/2410.14716/) (2024)
- [Multi-objective Evolution of Heuristic Using Large Language Model](https://mlsanity.com/paper/2409.16867/) (2024)
- [EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents](https://mlsanity.com/paper/2403.12014/) (2024)
- [Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoning](https://mlsanity.com/paper/2410.16162/) (2024)
- [RL-GPT: Integrating Reinforcement Learning and Code-as-policy](https://mlsanity.com/paper/2402.19299/) (2024)
- [A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learning](https://mlsanity.com/paper/2410.14660/) (2024)
- [A Comprehensive Survey on Multi-Agent Cooperative Decision-Making: Scenarios, Approaches, Challenges and Perspectives](https://mlsanity.com/paper/2503.13415/) (2025)
- [Odyssey: Empowering Minecraft Agents with Open-World Skills](https://mlsanity.com/paper/2407.15325/) (2024)
- [Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policy](https://mlsanity.com/paper/2502.19902/) (2025)
- [A Decision-Language Model (DLM) for Dynamic Restless Multi-Armed Bandit Tasks in Public Health](https://mlsanity.com/paper/2402.14807/) (2024)
- [Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agents](https://mlsanity.com/paper/2510.23691/) (2025)
- [iLLM-TSC: Integration reinforcement learning and large language model for traffic signal control policy improvement](https://mlsanity.com/paper/2407.06025/) (2024)
- [A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challenges](https://mlsanity.com/paper/2403.10249/) (2024)
- [CurricuLLM: Automatic Task Curricula Design for Learning Complex Robot Skills using Large Language Models](https://mlsanity.com/paper/2409.18382/) (2024)
- [Continuously Improving Mobile Manipulation with Autonomous Real-World RL](https://mlsanity.com/paper/2409.20568/) (2024)
- [A Survey on Large Language Models for Automated Planning](https://mlsanity.com/paper/2502.12435/) (2025)
- [Open-World Reinforcement Learning over Long Short-Term Imagination](https://mlsanity.com/paper/2410.03618/) (2024)
- [LAMARL: LLM-Aided Multi-Agent Reinforcement Learning for Cooperative Policy Generation](https://mlsanity.com/paper/2506.01538/) (2025)
- [Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedback](https://mlsanity.com/paper/2410.23022/) (2024)
- [On the Modeling Capabilities of Large Language Models for Sequential Decision Making](https://mlsanity.com/paper/2410.05656/) (2024)
- [CausalMACE: Causality Empowered Multi-Agents in Minecraft Cooperative Tasks](https://mlsanity.com/paper/2508.18797/) (2025)
- [DSADF: Thinking Fast and Slow for Decision Making](https://mlsanity.com/paper/2505.08189/) (2025)
- [Evaluating Vision-Language Models as Evaluators in Path Planning](https://mlsanity.com/paper/2411.18711/) (2024)
- [EscapeCraft: A 3D Room Escape Environment for Benchmarking Complex Multimodal Reasoning Ability](https://mlsanity.com/paper/2503.10042/) (2025)
- [Experience Transfer for Multimodal LLM Agents in Minecraft Game](https://mlsanity.com/paper/2604.05533/) (2026)
- [Language Models as Efficient Reward Function Searchers for Custom-Environment Multi-Objective Reinforcement](https://mlsanity.com/paper/2409.02428/) (2024)
- [Automated Rewards via LLM-Generated Progress Functions](https://mlsanity.com/paper/2410.09187/) (2024)
- [Large Language Model-Enhanced Reinforcement Learning for Generic Bus Holding Control Strategies](https://mlsanity.com/paper/2410.10212/) (2024)

## Abstract (arXiv)

Many reinforcement learning environments (e.g., Minecraft) provide only sparse rewards that indicate task completion or failure with binary values. The challenge in exploration efficiency in such environments makes it difficult for reinforcement-learning-based agents to learn complex tasks. To address this, this paper introduces an advanced learning system, named Auto MC-Reward, that leverages Large Language Models (LLMs) to automatically design dense reward functions, thereby enhancing the learning efficiency. Auto MC-Reward consists of three important components: Reward Designer, Reward Critic, and Trajectory Analyzer. Given the environment information and task descriptions, the Reward Designer first design the reward function by coding an executable Python function with predefined observation inputs. Then, our Reward Critic will be responsible for verifying the code, checking whether the code is self-consistent and free of syntax and semantic errors. Further, the Trajectory Analyzer summarizes possible failure causes and provides refinement suggestions according to collected trajectories. In the next round, Reward Designer will further refine and iterate the dense reward function based on feedback. Experiments demonstrate a significant improvement in the success rate and learning efficiency of our agents in complex tasks in Minecraft, such as obtaining diamond with the efficient ability to avoid lava, and efficiently explore trees and animals that are sparse in the plains biome.