# MaestroMotif: Skill Design from Artificial Intelligence Feedback

arXiv: https://arxiv.org/abs/2412.08542 · HTML: https://mlsanity.com/paper/2412.08542/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 63 references that are in the map.
- First posted on arXiv 11 Dec 2024.

## Builds on (most cited first, 30 of 36)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](https://mlsanity.com/paper/2304.11477/) (2023)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [Modular Multitask Reinforcement Learning with Policy Sketches](https://mlsanity.com/paper/1611.01796/) (2016)
- [The NetHack Learning Environment](https://mlsanity.com/paper/2006.13760/) (2020)
- [Goal-Conditioned Reinforcement Learning: Problems and Solutions](https://mlsanity.com/paper/2201.08299/) (2022)
- [Generalized Planning in PDDL Domains with Pretrained Large Language Models](https://mlsanity.com/paper/2305.11014/) (2023)
- [A Laplacian Framework for Option Discovery in Reinforcement Learning](https://mlsanity.com/paper/1703.00956/) (2017)
- [Motif: Intrinsic Motivation from Artificial Intelligence Feedback](https://mlsanity.com/paper/2310.00166/) (2023)
- [Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning](https://mlsanity.com/paper/2006.11751/) (2020)
- [Does Zero-Shot Reinforcement Learning Exist?](https://mlsanity.com/paper/2209.14935/) (2022)
- [Open-Endedness is Essential for Artificial Superhuman Intelligence](https://mlsanity.com/paper/2406.04268/) (2024)
- [When Waiting is not an Option : Learning Options with a Deliberation Cost](https://mlsanity.com/paper/1709.04571/) (2017)
- [Temporal Abstraction in Reinforcement Learning with the Successor Representation](https://mlsanity.com/paper/2110.05740/) (2021)
- [Exploration via Elliptical Episodic Bonuses](https://mlsanity.com/paper/2210.05805/) (2022)
- [Options of Interest: Temporal Abstraction with Interest Functions](https://mlsanity.com/paper/2001.00271/) (2020)

## Built on by (most cited first, 24 of 24)

- [Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents](https://mlsanity.com/paper/2505.22954/) (2025)
- [A Self-Improving Coding Agent](https://mlsanity.com/paper/2504.15228/) (2025)
- [LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities](https://mlsanity.com/paper/2504.16078/) (2025)
- [The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning](https://mlsanity.com/paper/2502.15214/) (2025)
- [One Life to Learn: Inferring Symbolic World Models for Stochastic Environments from Unguided Exploration](https://mlsanity.com/paper/2510.12088/) (2025)
- [Foundation Model Self-Play: Open-Ended Strategy Innovation via Foundation Models](https://mlsanity.com/paper/2507.06466/) (2025)
- [Scalable Option Learning in High-Throughput Environments](https://mlsanity.com/paper/2509.00338/) (2025)
- [ARM-FM: Automated Reward Machines via Foundation Models for Compositional Reinforcement Learning](https://mlsanity.com/paper/2510.14176/) (2025)
- [SCALAR: Learning and Composing Skills through LLM Guided Symbolic Planning and Deep RL Grounding](https://mlsanity.com/paper/2603.09036/) (2026)
- [Multi-Agent Transactive Memory](https://mlsanity.com/paper/2606.19911/) (2026)
- [Automated Reward Design for Gran Turismo](https://mlsanity.com/paper/2511.02094/) (2025)
- [Beyond Fixed Tasks: Unsupervised Environment Design for Task-Level Pairs](https://mlsanity.com/paper/2511.12706/) (2025)
- [Self-correcting Reward Shaping via Language Models for Reinforcement Learning Agents in Games](https://mlsanity.com/paper/2506.23626/) (2025)
- [SIDiffAgent: Self-Improving Diffusion Agent](https://mlsanity.com/paper/2602.02051/) (2026)
- [Zero-Shot Instruction Following in RL via Structured LTL Representations](https://mlsanity.com/paper/2602.14344/) (2026)
- [Agentick: A Unified Benchmark for General Sequential Decision-Making Agents](https://mlsanity.com/paper/2605.06869/) (2026)
- [Zero-Shot Instruction Following in RL via Structured LTL Representations](https://mlsanity.com/paper/2512.02633/) (2025)
- [Hierarchical Behaviour Spaces](https://mlsanity.com/paper/2604.24558/) (2026)
- [Goal-Conditioned Agents that Learn Everything All at Once](https://mlsanity.com/paper/2605.23551/) (2026)
- [Task diversity produces systematic transfer but inhibits continual reinforcement learning](https://mlsanity.com/paper/2606.00880/) (2026)
- [SCOPE: Synthetic Conditional Objectives for Policy Evolution in Black-Box Combinatorial Optimization](https://mlsanity.com/paper/2607.27630/) (2026)
- [Success Leaves Detours: Learning Executable Walkthroughs for Long-Horizon Agents](https://mlsanity.com/paper/2609.22120/) (2026)
- [Up and Down the Abstraction Ladder: Code-Based Skills for Language Agents](https://mlsanity.com/paper/2609.31076/) (2026)
- [Can Jev be Your Q or Policy in Reinforcement Learning?](https://mlsanity.com/paper/2610.11692/) (2026)

## Abstract (arXiv)

Describing skills in natural language has the potential to provide an accessible way to inject human knowledge about decision-making into an AI system. We present MaestroMotif, a method for AI-assisted skill design, which yields high-performing and adaptable agents. MaestroMotif leverages the capabilities of Large Language Models (LLMs) to effectively create and reuse skills. It first uses an LLM's feedback to automatically design rewards corresponding to each skill, starting from their natural language description. Then, it employs an LLM's code generation abilities, together with reinforcement learning, for training the skills and combining them to implement complex behaviors specified in language. We evaluate MaestroMotif using a suite of complex tasks in the NetHack Learning Environment (NLE), demonstrating that it surpasses existing approaches in both performance and usability.