mlsanityData to 14 Oct 2026

Program Machine Policy: Addressing Long-Horizon Tasks by Integrating Program Synthesis and State Machines

arXiv 2311.15960 ↗First posted 27 Nov 2023cs.LG cs.AI cs.PLcited by 1

Not on the map. Why?

Abstract (arXiv)

Deep reinforcement learning (deep RL) excels in various domains but lacks generalizability and interpretability. On the other hand, programmatic RL methods (Trivedi et al., 2021; Liu et al., 2023) reformulate RL tasks as synthesizing interpretable programs that can be executed in the environments. Despite encouraging results, these methods are limited to short-horizon tasks. On the other hand, representing RL policies using state machines (Inala et al., 2020) can inductively generalize to long-horizon tasks; however, it struggles to scale up to acquire diverse and complex behaviors. This work proposes the Program Machine Policy (POMP), which bridges the advantages of programmatic RL and state machine policies, allowing for the representation of complex behaviors and the address of long-term tasks. Specifically, we introduce a method that can retrieve a set of effective, diverse, and compatible programs. Then, we use these programs as modes of a state machine and learn a transition function to transition among mode programs, allowing for capturing repetitive behaviors. Our proposed framework outperforms programmatic RL and deep RL baselines on various tasks and demonstrates the ability to inductively generalize to even longer horizons without any fine-tuning. Ablation studies justify the effectiveness of our proposed search algorithm for retrieving a set of programs as modes.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 36 of its 73 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

36 of 73 references.

References · 36

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  3. 2016Generative Adversarial Imitation Learningcited by 1,597
  4. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  5. 2016The Mythos of Model Interpretabilitycited by 721
  6. 2016The Option-Critic Architecturecited by 448
  7. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
  8. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  9. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  10. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  11. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
  12. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 203
24 more
  1. 2018Programmatically Interpretable Reinforcement Learningcited by 193
  2. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 179
  3. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 177
  4. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 174
  5. 2020DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learningcited by 161
  6. 2016DeepCoder: Learning to Write Programscited by 149
  7. 2017RobustFill: Neural Program Learning under Noisy I/Ocited by 126
  8. 2017Meta Learning Shared Hierarchiescited by 93
  9. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 86
  10. 2018Leveraging Grammar and Reinforcement Learning for Neural Program Synthesiscited by 84
  11. 2021Learning to Synthesize Programs as Interpretable and Generalizable Policiescited by 59
  12. 2019Joint Inference of Reward Machines and Policies for Reinforcement Learningcited by 56
  13. 2018PEORL: Integrating Symbolic Planning and Hierarchical Reinforcement Learning for Robust Decision-Makingcited by 52
  14. 2023Demo2Code: From Summarizing Demonstrations to Synthesizing Code via Extended Chain-of-Thoughtcited by 39
  15. 2023Diffusion Model-Augmented Behavioral Cloningcited by 38
  16. 2018Learning Finite State Representations of Recurrent Policy Networkscited by 37
  17. 2022Skill-based Meta-Reinforcement Learningcited by 37
  18. 2022LEAGUE: Guided Skill Learning and Abstraction for Long-Horizon Manipulationcited by 34
  19. 2021Flexible Option Learningcited by 25
  20. 2022Leveraging Approximate Symbolic Models for Reinforcement Learning via Skill Diversitycited by 22
  21. 2023Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programscited by 20
  22. 2020Latent Programmer: Discrete Latent Codes for Program Synthesiscited by 18
  23. 2023Hierarchical Neural Program Synthesiscited by 7
  24. 2022Hierarchies of Reward Machinescited by 4

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025AI Agents as Universal Task Solverscited by 9