# NAPS: Natural Program Synthesis Dataset

arXiv: https://arxiv.org/abs/1807.03168 · HTML: https://mlsanity.com/paper/1807.03168/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jul 2018.

## Built on by (most cited first, 14 of 14)

- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [Language Models Can Teach Themselves to Program Better](https://mlsanity.com/paper/2207.14502/) (2022)
- [Execution-based Evaluation for Data Science Code Generation Models](https://mlsanity.com/paper/2211.09374/) (2022)
- [Fully Autonomous Programming with Large Language Models](https://mlsanity.com/paper/2304.10423/) (2023)
- [Programming Puzzles](https://mlsanity.com/paper/2106.05784/) (2021)
- [Efficacy of Various Large Language Models in Generating Smart Contracts](https://mlsanity.com/paper/2407.11019/) (2024)
- [Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Quality](https://mlsanity.com/paper/2503.05860/) (2025)
- [On Adversarial Robustness of Synthetic Code Generation](https://mlsanity.com/paper/2106.11629/) (2021)
- [A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspective](https://mlsanity.com/paper/2202.04847/) (2022)
- [RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in Romanian](https://mlsanity.com/paper/2402.13222/) (2024)
- [Fully Autonomous Programming using Iterative Multi-Agent Debugging with Large Language Models](https://mlsanity.com/paper/2503.07693/) (2025)
- [JEMMA: An Extensible Java Dataset for ML4Code Applications](https://mlsanity.com/paper/2212.09132/) (2022)
- [Generating Unseen Code Tests In Infinitum](https://mlsanity.com/paper/2407.19772/) (2024)

## Abstract (arXiv)

We present a program synthesis-oriented dataset consisting of human written problem statements and solutions for these problems. The problem statements were collected via crowdsourcing and the program solutions were extracted from human-written solutions in programming competitions, accompanied by input/output examples. We propose using this dataset for the program synthesis tasks aimed for working with real user-generated data. As a baseline we present few models, with the best model achieving 8.8% accuracy, showcasing both the complexity of the dataset and large room for future research.