# Generating Pragmatic Examples to Train Neural Program Synthesizers

arXiv: https://arxiv.org/abs/2311.05740 · HTML: https://mlsanity.com/paper/2311.05740/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 23 references that are in the map.
- First posted on arXiv 9 Nov 2023.

## Builds on (most cited first, 18 of 18)

- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Speaker-Follower Models for Vision-and-Language Navigation](https://mlsanity.com/paper/1806.02724/) (2018)
- [DeepCoder: Learning to Write Programs](https://mlsanity.com/paper/1611.01989/) (2016)
- [RobustFill: Neural Program Learning under Noisy I/O](https://mlsanity.com/paper/1703.07469/) (2017)
- [Reasoning About Pragmatics with Neural Listeners and Speakers](https://mlsanity.com/paper/1604.00562/) (2016)
- [Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learning](https://mlsanity.com/paper/2005.07064/) (2020)
- [Unified Pragmatic Models for Generating and Following Instructions](https://mlsanity.com/paper/1711.04987/) (2017)
- [Pragmatically Informative Image Captioning with Character-Level Inference](https://mlsanity.com/paper/1804.05417/) (2018)
- [SpreadsheetCoder: Formula Prediction from Semi-structured Context](https://mlsanity.com/paper/2106.15339/) (2021)
- [Program Synthesis using Conflict-Driven Learning](https://mlsanity.com/paper/1711.08029/) (2017)
- [Learning to refer informatively by amortizing pragmatic reasoning](https://mlsanity.com/paper/2006.00418/) (2020)
- [Learning in the Rational Speech Acts Model](https://mlsanity.com/paper/1510.06807/) (2015)
- [Learning to learn generative programs with Memoised Wake-Sleep](https://mlsanity.com/paper/2007.03132/) (2020)
- [Program Synthesis with Pragmatic Communication](https://mlsanity.com/paper/2007.05060/) (2020)
- [Lost in Machine Translation: A Method to Reduce Meaning Loss](https://mlsanity.com/paper/1902.09514/) (2019)
- [The Usability of Pragmatic Communication in Regular Expression Synthesis](https://mlsanity.com/paper/2308.06656/) (2023)
- [FOREST: An Interactive Multi-tree Synthesizer for Regular Expressions](https://mlsanity.com/paper/2012.14235/) (2020)

## Built on by (most cited first, 5 of 5)

- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges](https://mlsanity.com/paper/2502.12378/) (2025)
- [LACIE: Listener-Aware Finetuning for Confidence Calibration in Large Language Models](https://mlsanity.com/paper/2405.21028/) (2024)
- [Amortizing Pragmatic Program Synthesis with Rankings](https://mlsanity.com/paper/2407.02499/) (2024)
- [Identifying & Interactively Refining Ambiguous User Goals for Data Visualization Code Generation](https://mlsanity.com/paper/2510.09390/) (2025)

## Abstract (arXiv)

Programming-by-example is the task of synthesizing a program that is consistent with a set of user-provided input-output examples. As examples are often an under-specification of one's intent, a good synthesizer must choose the intended program from the many that are consistent with the given set of examples. Prior work frames program synthesis as a cooperative game between a listener (that synthesizes programs) and a speaker (a user choosing examples), and shows that models of computational pragmatic inference are effective in choosing the user intended programs. However, these models require counterfactual reasoning over a large set of programs and examples, which is infeasible in realistic program spaces. In this paper, we propose PraX, a novel way to amortize this search with neural networks. We sample pairs of programs and examples via self-play between listener and speaker models, and use pragmatic inference to choose informative training examples from this sample. We then use the informative dataset to train models to improve the synthesizer's ability to disambiguate user-provided examples without human supervision. We validate PraX on the challenging task of synthesizing regular expressions from example strings, and find that our method (1) outperforms models trained without choosing pragmatic examples by 23% (a 51% relative increase) (2) matches the performance of supervised learning on a dataset of pragmatic examples provided by humans, despite using no human data in training.