# InnateCoder: Learning Programmatic Options with Foundation Models

arXiv: https://arxiv.org/abs/2505.12508 · HTML: https://mlsanity.com/paper/2505.12508/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Interpretable and logic-guided RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 40 of its 58 references that are in the map.
- First posted on arXiv 18 May 2025.

## Placed in

- Deep reinforcement learning methods / [Interpretable and logic-guided RL](https://mlsanity.com/deeprl/interpretable-and-logic-guided-rl/)

## Builds on (most cited first, 30 of 40)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents](https://mlsanity.com/paper/2302.01560/) (2023)
- [Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory](https://mlsanity.com/paper/2305.17144/) (2023)
- [Towards Continual Reinforcement Learning: A Review and Perspectives](https://mlsanity.com/paper/2012.13490/) (2020)
- [A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning](https://mlsanity.com/paper/1711.00832/) (2017)
- [Transfer Learning in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2009.07888/) (2020)
- [Large Language Models as Commonsense Knowledge for Large-Scale Task Planning](https://mlsanity.com/paper/2305.14078/) (2023)
- [Programmatically Interpretable Reinforcement Learning](https://mlsanity.com/paper/1804.02477/) (2018)
- [Verifiable Reinforcement Learning via Policy Extraction](https://mlsanity.com/paper/1805.08328/) (2018)
- [Generalized Planning in PDDL Domains with Pretrained Large Language Models](https://mlsanity.com/paper/2305.11014/) (2023)
- [Evolution through Large Models](https://mlsanity.com/paper/2206.08896/) (2022)
- [EvoPrompting: Language Models for Code-Level Neural Architecture Search](https://mlsanity.com/paper/2302.14838/) (2023)
- [Large Language Models as Evolutionary Optimizers](https://mlsanity.com/paper/2310.19046/) (2023)
- [Variational Option Discovery Algorithms](https://mlsanity.com/paper/1807.10299/) (2018)
- [Language Model Crossover: Variation through Few-Shot Prompting](https://mlsanity.com/paper/2302.12170/) (2023)
- [A Laplacian Framework for Option Discovery in Reinforcement Learning](https://mlsanity.com/paper/1703.00956/) (2017)
- [Motif: Intrinsic Motivation from Artificial Intelligence Feedback](https://mlsanity.com/paper/2310.00166/) (2023)
- [Meta Learning Shared Hierarchies](https://mlsanity.com/paper/1710.09767/) (2017)
- [A Deep Hierarchical Approach to Lifelong Learning in Minecraft](https://mlsanity.com/paper/1604.07255/) (2016)
- [LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimization](https://mlsanity.com/paper/2306.01102/) (2023)
- [Eigenoption Discovery through the Deep Successor Representation](https://mlsanity.com/paper/1710.11089/) (2017)
- [Learning to Synthesize Programs as Interpretable and Generalizable Policies](https://mlsanity.com/paper/2108.13643/) (2021)
- [Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots](https://mlsanity.com/paper/1301.4862/) (2013)
- [Towards Optimizing with Large Language Models](https://mlsanity.com/paper/2310.05204/) (2023)
- [Deep Laplacian-based Options for Temporally-Extended Exploration](https://mlsanity.com/paper/2301.11181/) (2023)

## Built on by (most cited first, 2 of 2)

- [Scheduling That Speaks: An Interpretable Programmatic Reinforcement Learning Framework](https://mlsanity.com/paper/2605.18454/) (2026)
- [Constrained Command-Conditioned Reinforcement Learning with Bandit Strategy Selection in Real-Time Strategy Games](https://mlsanity.com/paper/2610.11663/) (2026)

## Abstract (arXiv)

Outside of transfer learning settings, reinforcement learning agents start their learning process from a clean slate. As a result, such agents have to go through a slow process to learn even the most obvious skills required to solve a problem. In this paper, we present InnateCoder, a system that leverages human knowledge encoded in foundation models to provide programmatic policies that encode "innate skills" in the form of temporally extended actions, or options. In contrast to existing approaches to learning options, InnateCoder learns them from the general human knowledge encoded in foundation models in a zero-shot setting, and not from the knowledge the agent gains by interacting with the environment. Then, InnateCoder searches for a programmatic policy by combining the programs encoding these options into larger and more complex programs. We hypothesized that InnateCoder's way of learning and using options could improve the sampling efficiency of current methods for learning programmatic policies. Empirical results in MicroRTS and Karel the Robot support our hypothesis, since they show that InnateCoder is more sample efficient than versions of the system that do not use options or learn them from experience.