mlsanityData to 14 Oct 2026

Synthesizing Programmatic Reinforcement Learning Policies with Large Language Model Guided Search

arXiv 2405.16450 ↗First posted 26 May 2024cs.LG cs.AI cs.PLcited by 13

Not on the map. Why?

Abstract (arXiv)

Programmatic reinforcement learning (PRL) has been explored for representing policies through programs as a means to achieve interpretability and generalization. Despite promising outcomes, current state-of-the-art PRL methods are hindered by sample inefficiency, necessitating tens of millions of program-environment interactions. To tackle this challenge, we introduce a novel LLM-guided search framework (LLM-GS). Our key insight is to leverage the programming expertise and common sense reasoning of LLMs to enhance the efficiency of assumption-free, random-guessing search methods. We address the challenge of LLMs' inability to generate precise and grammatically correct programs in domain-specific languages (DSLs) by proposing a Pythonic-DSL strategy - an LLM is instructed to initially generate Python codes and then convert them into DSL programs. To further optimize the LLM-generated programs, we develop a search algorithm named Scheduled Hill Climbing, designed to efficiently explore the programmatic search space to improve the programs consistently. Experimental results in the Karel domain demonstrate our LLM-GS framework's superior effectiveness and efficiency. Extensive ablation studies further verify the critical role of our Pythonic-DSL strategy and Scheduled Hill Climbing algorithm. Moreover, we conduct experiments with two novel tasks, showing that LLM-GS enables users without programming skills and knowledge of the domain or DSL to describe the tasks in natural language to obtain performant programs.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 49 of its 96 references that are in the map. First posted on arXiv 26 May 2024.

Builds on

49 of 96 references.

References · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  8. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  9. 2017Hindsight Experience Replaycited by 1,136
  10. 2023Teaching Large Language Models to Self-Debugcited by 988
  11. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  12. 2022Galactica: A Large Language Model for Sciencecited by 744
37 more
  1. 2019Model-Based Reinforcement Learning for Ataricited by 502
  2. 2022A Systematic Evaluation of Large Language Models of Codecited by 363
  3. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  4. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  5. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 197
  6. 2018Programmatically Interpretable Reinforcement Learningcited by 193
  7. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 177
  8. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  9. 2023Take a Step Back: Evoking Reasoning via Abstraction in Large Language Modelscited by 159
  10. 2016DeepCoder: Learning to Write Programscited by 149
  11. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environmentcited by 107
  12. 2018NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating Systemcited by 98
  13. 2020Explainability in Deep Reinforcement Learningcited by 97
  14. 2016Neuro-Symbolic Program Synthesiscited by 84
  15. 2018Leveraging Grammar and Reinforcement Learning for Neural Program Synthesiscited by 84
  16. 2021Learning to Synthesize Programs as Interpretable and Generalizable Policiescited by 59
  17. 2023Grammar Prompting for Domain-Specific Language Generation with Large Language Modelscited by 58
  18. 2019Neural Logic Reinforcement Learningcited by 53
  19. 2024Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformerscited by 52
  20. 2023LILO: Learning Interpretable Libraries by Compressing and Documenting Codecited by 49
  21. 2023Interpretable and Explainable Logical Policies via Neurally Guided Symbolic Abstractioncited by 46
  22. 2015Program Synthesis using Natural Languagecited by 41
  23. 2019An Inductive Synthesis Framework for Verifiable Reinforcement Learningcited by 39
  24. 2020Synthesize, Execute and Debug: Learning to Repair for Neural Program Synthesiscited by 39
  25. 2018Learning Finite State Representations of Recurrent Policy Networkscited by 37
  26. 2023Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priorscited by 36
  27. 2021Latent Execution for Neural Program Synthesiscited by 35
  28. 2019Few-Shot Bayesian Imitation Learning with Logical Program Policiescited by 30
  29. 2024CodeIt: Self-Improving Language Models with Prioritized Hindsight Replaycited by 23
  30. 2017Naturalizing a Programming Language via Interactive Learningcited by 20
  31. 2023Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programscited by 20
  32. 2024Pix2Code: Learning to Compose Neural Visual Concepts as Programscited by 18
  33. 2024Interpretable and Editable Programmatic Tree Policies for Reinforcement Learningcited by 18
  34. 2024Leveraging Print Debugging to Improve Code Generation in Large Language Modelscited by 15
  35. 2023Learning of Generalizable and Interpretable Knowledge in Grid-Based Reinforcement Learning Environmentscited by 5
  36. 2024Searching for Programmatic Policies in Semantic Spacescited by 5
  37. 2018Towards Mixed Optimization for Reinforcement Learning with Program Synthesiscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 5.

13 papers.

Sort:

By area:

  1. 2025Reward Is Enough: LLMs Are In-Context Reinforcement Learnerscited by 31
  2. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Modelscited by 21
  3. 2025Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learningcited by 14
  4. 2025BILLY: Steering Large Language Models via Merging Persona Vectors for Creative Generationcited by 11
  5. 2025Multimodal LLM-assisted Evolutionary Search for Programmatic Control Policiescited by 8
  6. 2024From Code to Play: Benchmarking Program Search for Games Using Large Language Modelscited by 5
  7. 2026Janus: An Algorithm-Evaluator Co-Evolution Framework for LLM-Driven Discovery under Expensive Evaluation Budgetscited by 3
  8. 2026REFLEX: Reflective Evolution from LLM Experiencecited by 2
  9. 2025PolicyEvolve: Evolving Programmatic Policies by LLMs for multi-player games via Population-Based Trainingcited by 1
  10. 2025Policy-Conditioned Policies for Multi-Agent Task Solvingcited by 0
  11. 2026DiPRL: Learning Discrete Programmatic Policies via Architecture Entropy Regularizationcited by 0
  12. 2026Language-Critique Imitation Learning from Suboptimal Demonstrationscited by 0
1 more
  1. 2026Synthesizing Reactive Character Behaviors for Continuous Games via Programmatic Policy Searchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs
  2. 2026Enhanced LLM Reasoning by Optimizing Reward Functions with Search-Driven Reinforcement Learning
  3. 2025Algorithm Discovery With LLMs: Evolutionary Search Meets Reinforcement Learning
  4. 2025LLM-Guided Reinforcement Learning: Addressing Training Bottlenecks through Policy Modulation
  5. 2025Guiding Exploration in Reinforcement Learning Through LLM-Augmented Observations
  6. 2024How Can LLM Guide RL? A Value-Based Approach
  7. 2026LLM-assisted Semantic Option Discovery for Facilitating Adaptive Deep Reinforcement Learning
  8. 2025LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models