mlsanityData to 7 Oct 2026

Program Synthesis Guided Reinforcement Learning for Partially Observed Environments

arXiv 2102.11137 ↗First posted 22 Feb 2021cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

A key challenge for reinforcement learning is solving long-horizon planning problems. Recent work has leveraged programs to guide reinforcement learning in these settings. However, these approaches impose a high manual burden on the user since they must provide a guiding program for every new task. Partially observed environments further complicate the programming task because the program must implement a strategy that correctly, and ideally optimally, handles every possible configuration of the hidden regions of the environment. We propose a new approach, model predictive program synthesis (MPPS), that uses program synthesis to automatically generate the guiding programs. It trains a generative model to predict the unobserved portions of the world, and then synthesizes a program based on samples from this model in a way that is robust to its uncertainty. In our experiments, we show that our approach significantly outperforms non-program-guided approaches on a set of challenging benchmarks, including a 2D Minecraft-inspired environment where the agent must complete a complex sequence of subtasks to achieve its goal, and achieves a similar performance as using handcrafted programs to guide the agent. Our results demonstrate that our approach can obtain the benefits of program-guided reinforcement learning without requiring the user to provide a new guiding program for every new task.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 39 of its 80 references that are in the map. First posted on arXiv 22 Feb 2021.

Builds on

39 of 80 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2013Auto-Encoding Variational Bayescited by 3,980
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  7. 2018World Modelscited by 1,444
  8. 2017A Brief Survey of Deep Reinforcement Learningcited by 466
  9. 2020Integrated Task and Motion Planningcited by 281
  10. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 200
  11. 2018Programmatically Interpretable Reinforcement Learningcited by 192
  12. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 176
27 more
  1. 2016DeepCoder: Learning to Write Programscited by 145
  2. 2017RobustFill: Neural Program Learning under Noisy I/Ocited by 124
  3. 2019Write, Execute, Assess: Program Synthesis with a REPLcited by 92
  4. 2016A Deep Hierarchical Approach to Lifelong Learning in Minecraftcited by 90
  5. 2018Leveraging Grammar and Reinforcement Learning for Neural Program Synthesiscited by 83
  6. 2017Learning to Infer Graphics Programs from Hand-Drawn Imagescited by 73
  7. 2019Learning to Infer and Execute 3D Shape Programscited by 68
  8. 2020A Composable Specification Language for Reinforcement Learning Taskscited by 68
  9. 2021Hierarchical Reinforcement Learning By Discovering Intrinsic Optionscited by 57
  10. 2020Neurosymbolic Reinforcement Learning with Formally Verified Explorationcited by 49
  11. 2019Sub-policy Adaptation for Hierarchical Reinforcement Learningcited by 47
  12. 2018Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependenciescited by 43
  13. 2019DAC: The Double Actor-Critic Architecture for Learning Optionscited by 43
  14. 2017Learning Generalized Reactive Policies using Deep Neural Networkscited by 39
  15. 2018Neural-Guided Deductive Search for Real-Time Program Synthesis from Examplescited by 39
  16. 2018HOUDINI: Lifelong Learning as Program Synthesiscited by 38
  17. 2020Data-efficient Hindsight Off-policy Option Learningcited by 31
  18. 2020Learning Differentiable Programs with Admissible Neural Heuristicscited by 30
  19. 2017Program Synthesis using Conflict-Driven Learningcited by 26
  20. 2020Representing Partial Programs with Blended Abstract Semanticscited by 14
  21. 2021Neurosymbolic Transformers for Multi-Agent Communicationcited by 13
  22. 2020Abstract Value Iteration for Hierarchical Reinforcement Learningcited by 12
  23. 2019Learning Neurosymbolic Generative Models via Program Synthesiscited by 11
  24. 2019DeepSynth: Automata Synthesis for Automatic Task Segmentation in Deep Reinforcement Learningcited by 8
  25. 2020Inductive Program Synthesis Over Noisy Datacited by 7
  26. 2018Neural Guided Constraint Logic Programming for Program Synthesiscited by 6
  27. 2017Synthesis of Data Completion Scripts using Finite Tree Automatacited by 4

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 3; 2023: 1.

7 papers.

  1. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  2. 2021Learning to Synthesize Programs as Interpretable and Generalizable Policiescited by 59
  3. 2022TASKOGRAPHY: Evaluating robot task planning over large 3D scene graphscited by 59
  4. 2021Programmatic Reward Design by Examplecited by 9
  5. 2022A Hierarchical Bayesian Approach to Inverse Reinforcement Learning with Symbolic Reward Machinescited by 9
  6. 2022Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templatescited by 2
  7. 2023Improving Reliable Navigation under Uncertainty via Predictions Informed by Non-Local Informationcited by 0