# What can we Learn Even From the Weakest? Learning Sketches for Programmatic Strategies

arXiv: https://arxiv.org/abs/2203.11912 · HTML: https://mlsanity.com/paper/2203.11912/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 6 of its 27 references that are in the map.
- First posted on arXiv 22 Mar 2022.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 6 of 6)

- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning](https://mlsanity.com/paper/1711.00832/) (2017)
- [Programmatically Interpretable Reinforcement Learning](https://mlsanity.com/paper/1804.02477/) (2018)
- [Verifiable Reinforcement Learning via Policy Extraction](https://mlsanity.com/paper/1805.08328/) (2018)
- [Learning to Infer Program Sketches](https://mlsanity.com/paper/1902.06349/) (2019)
- [Evolving Agents for the Hanabi 2018 CIG Competition](https://mlsanity.com/paper/1809.09764/) (2018)

## Built on by (most cited first, 11 of 11)

- [Choosing Well Your Opponents: How to Guide the Synthesis of Programmatic Strategies](https://mlsanity.com/paper/2307.04893/) (2023)
- [Searching for Programmatic Policies in Semantic Spaces](https://mlsanity.com/paper/2405.05431/) (2024)
- [Programmatic Imitation Learning from Unlabeled and Noisy Demonstrations](https://mlsanity.com/paper/2303.01440/) (2023)
- [Reinforcement Learning and Data-Generation for Syntax-Guided Synthesis](https://mlsanity.com/paper/2307.09564/) (2023)
- [Synthesizing Programmatic Policies with Actor-Critic Algorithms and ReLU Networks](https://mlsanity.com/paper/2308.02729/) (2023)
- [Assessing the Interpretability of Programmatic Policies with Large Language Models](https://mlsanity.com/paper/2311.06979/) (2023)
- [KnowPC: Knowledge-Driven Programmatic Reinforcement Learning for Zero-shot Coordination](https://mlsanity.com/paper/2408.04336/) (2024)
- [Reclaiming the Source of Programmatic Policies: Programmatic versus Latent Spaces](https://mlsanity.com/paper/2410.12166/) (2024)
- [InnateCoder: Learning Programmatic Options with Foundation Models](https://mlsanity.com/paper/2505.12508/) (2025)
- [Can You Improve My Code? Optimizing Programs with Local Search](https://mlsanity.com/paper/2307.05603/) (2023)
- [Learning Explainable Representations of Complex Game-playing Strategies](https://mlsanity.com/paper/2610.07638/) (2026)

## Abstract (arXiv)

In this paper we show that behavioral cloning can be used to learn effective sketches of programmatic strategies. We show that even the sketches learned by cloning the behavior of weak players can help the synthesis of programmatic strategies. This is because even weak players can provide helpful information, e.g., that a player must choose an action in their turn of the game. If behavioral cloning is not employed, the synthesizer needs to learn even the most basic information by playing the game, which can be computationally expensive. We demonstrate empirically the advantages of our sketch-learning approach with simulated annealing and UCT synthesizers. We evaluate our synthesizers in the games of Can't Stop and MicroRTS. The sketch-based synthesizers are able to learn stronger programmatic strategies than their original counterparts. Our synthesizers generate strategies of Can't Stop that defeat a traditional programmatic strategy for the game. They also synthesize strategies that defeat the best performing method from the latest MicroRTS competition.