# Towards Mixed Optimization for Reinforcement Learning with Program Synthesis

arXiv: https://arxiv.org/abs/1807.00403 · HTML: https://mlsanity.com/paper/1807.00403/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jul 2018.

## Built on by (most cited first, 4 of 4)

- [Synthesizing Programmatic Reinforcement Learning Policies with Large Language Model Guided Search](https://mlsanity.com/paper/2405.16450/) (2024)
- [XAI-N: Sensor-based Robot Navigation using Expert Policies and Decision Trees](https://mlsanity.com/paper/2104.10818/) (2021)
- [MSVIPER: Improved Policy Distillation for Reinforcement-Learning-Based Robot Navigation](https://mlsanity.com/paper/2209.09079/) (2022)
- [Bayesian Inverse Physics for Neuro-Symbolic Robot Learning](https://mlsanity.com/paper/2506.08756/) (2025)

## Abstract (arXiv)

Deep reinforcement learning has led to several recent breakthroughs, though the learned policies are often based on black-box neural networks. This makes them difficult to interpret and to impose desired specification constraints during learning. We present an iterative framework, MORL, for improving the learned policies using program synthesis. Concretely, we propose to use synthesis techniques to obtain a symbolic representation of the learned policy, which can then be debugged manually or automatically using program repair. After the repair step, we use behavior cloning to obtain the policy corresponding to the repaired program, which is then further improved using gradient descent. This process continues until the learned policy satisfies desired constraints. We instantiate MORL for the simple CartPole problem and show that the programmatic representation allows for high-level modifications that in turn lead to improved learning of the policies.