mlsanityData to 14 Oct 2026

InnateCoder: Learning Programmatic Options with Foundation Models

arXiv 2505.12508 ↗First posted 18 May 2025cs.LGcited by 2

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Outside of transfer learning settings, reinforcement learning agents start their learning process from a clean slate. As a result, such agents have to go through a slow process to learn even the most obvious skills required to solve a problem. In this paper, we present InnateCoder, a system that leverages human knowledge encoded in foundation models to provide programmatic policies that encode "innate skills" in the form of temporally extended actions, or options. In contrast to existing approaches to learning options, InnateCoder learns them from the general human knowledge encoded in foundation models in a zero-shot setting, and not from the knowledge the agent gains by interacting with the environment. Then, InnateCoder searches for a programmatic policy by combining the programs encoding these options into larger and more complex programs. We hypothesized that InnateCoder's way of learning and using options could improve the sampling efficiency of current methods for learning programmatic policies. Empirical results in MicroRTS and Karel the Robot support our hypothesis, since they show that InnateCoder is more sample efficient than versions of the system that do not use options or learn them from experience.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 40 of its 58 references that are in the map. First posted on arXiv 18 May 2025.

Builds on

40 of 58 references.

References · 40

  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  3. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  4. 2023Large Language Models as Optimizerscited by 864
  5. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  6. 2016The Option-Critic Architecturecited by 448
  7. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 398
  8. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 289
  9. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  10. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 268
  11. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 265
  12. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
28 more
  1. 2018Programmatically Interpretable Reinforcement Learningcited by 193
  2. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 177
  3. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  4. 2022Evolution through Large Modelscited by 134
  5. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 131
  6. 2023Large Language Models as Evolutionary Optimizerscited by 124
  7. 2018Variational Option Discovery Algorithmscited by 122
  8. 2023Language Model Crossover: Variation through Few-Shot Promptingcited by 122
  9. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 110
  10. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 102
  11. 2017Meta Learning Shared Hierarchiescited by 93
  12. 2016A Deep Hierarchical Approach to Lifelong Learning in Minecraftcited by 90
  13. 2023LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimizationcited by 87
  14. 2017Eigenoption Discovery through the Deep Successor Representationcited by 86
  15. 2021Learning to Synthesize Programs as Interpretable and Generalizable Policiescited by 59
  16. 2013Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robotscited by 58
  17. 2023Towards Optimizing with Large Language Modelscited by 35
  18. 2023Deep Laplacian-based Options for Temporally-Extended Explorationcited by 29
  19. 2023Large Language Model for Multi-objective Evolutionary Optimizationcited by 29
  20. 2023Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programscited by 20
  21. 2021Gym-μRTS: Toward Affordable Full Game Real-time Strategy Games Research with Deep Reinforcement Learningcited by 19
  22. 2019Multitask Soft Option Learningcited by 13
  23. 2022What can we Learn Even From the Weakest? Learning Sketches for Programmatic Strategiescited by 11
  24. 2023Choosing Well Your Opponents: How to Guide the Synthesis of Programmatic Strategiescited by 8
  25. 2017Combinatorial Multi-armed Bandits for Real-Time Strategy Gamescited by 7
  26. 2024Searching for Programmatic Policies in Semantic Spacescited by 5
  27. 2024A Competition Winning Deep Reinforcement Learning Agent in microRTScited by 2
  28. 2024Reclaiming the Source of Programmatic Policies: Programmatic versus Latent Spacescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Scheduling That Speaks: An Interpretable Programmatic Reinforcement Learning Frameworkcited by 0
  2. 2026Constrained Command-Conditioned Reinforcement Learning with Bandit Strategy Selection in Real-Time Strategy Gamescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills
  2. 2026Better, Faster, Stronger: Programmatic Skill Learning Best Reduces Agent Cost
  3. 2024EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data
  4. 2026Up and Down the Abstraction Ladder: Code-Based Skills for Language Agents
  5. 2023LiFT: Unsupervised Reinforcement Learning with Foundation Models as Teachers
  6. 2026CODE-SHARP: Continuous Open-ended Discovery and Evolution of Skills as Hierarchical Reward Programs
  7. 2026OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning
  8. 2026Flow Policies as Actions of Skill-Level World Models: Learned and Symbolic Abstractions for Long-Horizon Planning