# Choosing Well Your Opponents: How to Guide the Synthesis of Programmatic Strategies

arXiv: https://arxiv.org/abs/2307.04893 · HTML: https://mlsanity.com/paper/2307.04893/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 28 references that are in the map.
- First posted on arXiv 10 Jul 2023.

## Builds on (most cited first, 7 of 7)

- [A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning](https://mlsanity.com/paper/1711.00832/) (2017)
- [Programmatically Interpretable Reinforcement Learning](https://mlsanity.com/paper/1804.02477/) (2018)
- [Verifiable Reinforcement Learning via Policy Extraction](https://mlsanity.com/paper/1805.08328/) (2018)
- [A Generalized Training Approach for Multiagent Learning](https://mlsanity.com/paper/1909.12823/) (2019)
- [Learning to Play No-Press Diplomacy with Best Response Policy Iteration](https://mlsanity.com/paper/2006.04635/) (2020)
- [What can we Learn Even From the Weakest? Learning Sketches for Programmatic Strategies](https://mlsanity.com/paper/2203.11912/) (2022)
- [Combinatorial Multi-armed Bandits for Real-Time Strategy Games](https://mlsanity.com/paper/1710.04805/) (2017)

## Built on by (most cited first, 8 of 8)

- [Searching for Programmatic Policies in Semantic Spaces](https://mlsanity.com/paper/2405.05431/) (2024)
- [From Code to Play: Benchmarking Program Search for Games Using Large Language Models](https://mlsanity.com/paper/2412.04057/) (2024)
- [Assessing the Interpretability of Programmatic Policies with Large Language Models](https://mlsanity.com/paper/2311.06979/) (2023)
- [A Competition Winning Deep Reinforcement Learning Agent in microRTS](https://mlsanity.com/paper/2402.08112/) (2024)
- [KnowPC: Knowledge-Driven Programmatic Reinforcement Learning for Zero-shot Coordination](https://mlsanity.com/paper/2408.04336/) (2024)
- [Reclaiming the Source of Programmatic Policies: Programmatic versus Latent Spaces](https://mlsanity.com/paper/2410.12166/) (2024)
- [InnateCoder: Learning Programmatic Options with Foundation Models](https://mlsanity.com/paper/2505.12508/) (2025)
- [DiPRL: Learning Discrete Programmatic Policies via Architecture Entropy Regularization](https://mlsanity.com/paper/2605.18508/) (2026)

## Abstract (arXiv)

This paper introduces Local Learner (2L), an algorithm for providing a set of reference strategies to guide the search for programmatic strategies in two-player zero-sum games. Previous learning algorithms, such as Iterated Best Response (IBR), Fictitious Play (FP), and Double-Oracle (DO), can be computationally expensive or miss important information for guiding search algorithms. 2L actively selects a set of reference strategies to improve the search signal. We empirically demonstrate the advantages of our approach while guiding a local search algorithm for synthesizing strategies in three games, including MicroRTS, a challenging real-time strategy game. Results show that 2L learns reference strategies that provide a stronger search signal than IBR, FP, and DO. We also simulate a tournament of MicroRTS, where a synthesizer using 2L outperformed the winners of the two latest MicroRTS competitions, which were programmatic strategies written by human programmers.