mlsanityData to 7 Oct 2026

Learning to Synthesize Programs as Interpretable and Generalizable Policies

arXiv 2108.13643 ↗First posted 31 Aug 2021cs.LG cs.AI cs.PLcited by 59

Not on the map. Why?

Abstract (arXiv)

Recently, deep reinforcement learning (DRL) methods have achieved impressive performance on tasks in a variety of domains. However, neural network policies produced with DRL methods are not human-interpretable and often have difficulty generalizing to novel scenarios. To address these issues, prior works explore learning programmatic policies that are more interpretable and structured for generalization. Yet, these works either employ limited policy representations (e.g. decision trees, state machines, or predefined program templates) or require stronger supervision (e.g. input/output state pairs or expert demonstrations). We present a framework that instead learns to synthesize a program, which details the procedure to solve a task in a flexible and expressive manner, solely from reward signals. To alleviate the difficulty of learning to compose programs to induce the desired agent behavior from scratch, we propose to first learn a program embedding space that continuously parameterizes diverse behaviors in an unsupervised manner and then search over the learned program embedding space to yield a program that maximizes the return for a given task. Experimental results demonstrate that the proposed framework not only learns to reliably synthesize task-solving programs but also outperforms DRL and program synthesis baselines while producing interpretable and more generalizable policies. We also justify the necessity of the proposed two-stage learning scheme as well as analyze various methods for learning the program embedding.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 70 of its 126 references that are in the map. First posted on arXiv 31 Aug 2021.

Builds on

70 of 126 references.

References · 70

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2013Auto-Encoding Variational Bayescited by 3,980
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  9. 2016Matching Networks for One Shot Learningcited by 2,246
  10. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  11. 2016The Mythos of Model Interpretabilitycited by 716
  12. 2019A Closer Look at Few-shot Classificationcited by 679
38 more
  1. 2018Learning Dexterous In-Hand Manipulationcited by 643
  2. 2014Neural Turing Machinescited by 630
  3. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  4. 2017Deep Learning for Medical Image Analysiscited by 399
  5. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  6. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  7. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  8. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  9. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 200
  10. 2018Programmatically Interpretable Reinforcement Learningcited by 192
  11. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 176
  12. 2020DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learningcited by 159
  13. 2017A General Safety Framework for Learning-Based Control in Uncertain Robotic Systemscited by 150
  14. 2016DeepCoder: Learning to Write Programscited by 145
  15. 2017RobustFill: Neural Program Learning under Noisy I/Ocited by 124
  16. 2015Neural GPUs Learn Algorithmscited by 118
  17. 2020Explainable deep learning models in medical image analysiscited by 111
  18. 2019Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulationcited by 103
  19. 2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 97
  20. 2018NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating Systemcited by 97
  21. 2015Neural Programmer-Interpreterscited by 96
  22. 2020Graph-based, Self-Supervised Program Repair from Diagnostic Feedbackcited by 93
  23. 2019Write, Execute, Assess: Program Synthesis with a REPLcited by 92
  24. 2017Cautious Model Predictive Control using Gaussian Process Regressioncited by 91
  25. 2016Neuro-Symbolic Program Synthesiscited by 84
  26. 2018Leveraging Grammar and Reinforcement Learning for Neural Program Synthesiscited by 83
  27. 2017Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learningcited by 77
  28. 2017Neural Task Programming: Learning to Generalize Across Hierarchical Taskscited by 75
  29. 2018Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstrationcited by 60
  30. 2020Estimating Uncertainty and Interpretability in Deep Learning for Coronavirus (COVID-19) Detectioncited by 46
  31. 2021Leveraging Language to Learn Program Abstractions and Search Heuristicscited by 44
  32. 2018Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependenciescited by 43
  33. 2020Cautious Adaptation For Reinforcement Learning in Safety-Critical Settingscited by 42
  34. 2021SpreadsheetCoder: Formula Prediction from Semi-structured Contextcited by 41
  35. 2019An Inductive Synthesis Framework for Verifiable Reinforcement Learningcited by 39
  36. 2020Neural Execution Engines: Learning to Execute Subroutinescited by 36
  37. 2017Making Neural Programming Architectures Generalize via Recursioncited by 35
  38. 2021Latent Execution for Neural Program Synthesiscited by 35

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 16; 2024: 17; 2025: 9; 2026: 11.

59 papers.

Sort:

By area:

  1. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  2. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  3. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  4. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  5. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  6. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 85
  7. 2024Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Searchcited by 38
  8. 2024RL-GPT: Integrating Reinforcement Learning and Code-as-policycited by 32
  9. 2024TwoStep: Multi-agent Task Planning using Classical Planners and Large Language Modelscited by 29
  10. 2023Neurosymbolic Reinforcement Learning and Planning: A Surveycited by 24
  11. 2024Interpretable Concept Bottlenecks to Align Reinforcement Learning Agentscited by 23
  12. 2023Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programscited by 20
38 more
  1. 2021ProTo: Program-Guided Transformer for Program-Guided Taskscited by 19
  2. 2024Pix2Code: Learning to Compose Neural Visual Concepts as Programscited by 18
  3. 2024Synthesizing Programmatic Reinforcement Learning Policies with Large Language Model Guided Searchcited by 13
  4. 2024A Survey on Verification and Validation, Testing and Evaluations of Neurosymbolic Artificial Intelligencecited by 10
  5. 2023Embodied Concept Learner: Self-supervised Learning of Concepts and Mapping through Instruction Followingcited by 9
  6. 2023Efficient Symbolic Policy Learning with Differentiable Symbolic Expressioncited by 9
  7. 2025One Life to Learn: Inferring Symbolic World Models for Stochastic Environments from Unguided Explorationcited by 9
  8. 2022From {Solution Synthesis} to {Student Attempt Synthesis} for Block-Based Visual Programming Taskscited by 8
  9. 2024Language Models can Infer Action Semantics for Symbolic Planners from Environment Feedbackcited by 8
  10. 2025Multimodal LLM-assisted Evolutionary Search for Programmatic Control Policiescited by 8
  11. 2025Learning Game-Playing Agents with Generative Code Optimizationcited by 8
  12. 2023Hierarchical Neural Program Synthesiscited by 7
  13. 2025Modeling Others' Minds as Codecited by 6
  14. 2023Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Surveycited by 5
  15. 2023Learning of Generalizable and Interpretable Knowledge in Grid-Based Reinforcement Learning Environmentscited by 5
  16. 2024Searching for Programmatic Policies in Semantic Spacescited by 5
  17. 2024Mitigating Information Loss in Tree-Based Reinforcement Learning via Direct Optimizationcited by 5
  18. 2026Code Evolution for Control: Synthesizing Policies via LLM-Driven Evolutionary Searchcited by 4
  19. 2022A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challengescited by 3
  20. 2023Programmatic Imitation Learning from Unlabeled and Noisy Demonstrationscited by 3
  21. 2025Evaluating Interpretable Reinforcement Learning by Distilling Policies into Programscited by 3
  22. 2026TIDE: Tuning-Integrated Dynamic Evolution for LLM-Based Automated Heuristic Designcited by 3
  23. 2026Safe and Interpretable Multimodal Path Planning for Multi-Agent Cooperationcited by 3
  24. 2023Pushing the Limits of Machine Design: Automated CPU Design with AIcited by 2
  25. 2024KnowPC: Knowledge-Driven Programmatic Reinforcement Learning for Zero-shot Coordinationcited by 2
  26. 2024Reclaiming the Source of Programmatic Policies: Programmatic versus Latent Spacescited by 2
  27. 2025InnateCoder: Learning Programmatic Options with Foundation Modelscited by 2
  28. 2023Optimizing Large Language Models to Expedite the Development of Smart Contractscited by 1
  29. 2023Learning a Hierarchical Planner from Humans in Multiple Generationscited by 1
  30. 2023Program Machine Policy: Addressing Long-Horizon Tasks by Integrating Program Synthesis and State Machinescited by 1
  31. 2024Three Pathways to Neurosymbolic Reinforcement Learning with Interpretable Model and Policy Networkscited by 1
  32. 2024Unveiling the Decision-Making Process in Reinforcement Learning with Genetic Programmingcited by 1
  33. 2024Human-Readable Programs as Actors of Reinforcement Learning Agents Using Critic-Moderated Evolutioncited by 1
  34. 2025PSALM-V: Automating Symbolic Planning in Interactive Visual Environments with Large Language Modelscited by 1
  35. 2026COvolve: Adversarial Co-Evolution of Large-Language-Model-Generated Policies and Environments via Two-Player Zero-Sum Gamecited by 1
  36. 2026REFLEX: Reflective Evolution from LLM Experiencecited by 1
  37. 2022Iterative Genetic Improvement: Scaling Stochastic Program Synthesiscited by 0
  38. 2023Worrisome Properties of Neural Network Controllers and Their Symbolic Representationscited by 0

50 most cited of 59 All 59 (JSON)