# Solving Probability and Statistics Problems by Program Synthesis

arXiv: https://arxiv.org/abs/2111.08267 · HTML: https://mlsanity.com/paper/2111.08267/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 14 references that are in the map.
- First posted on arXiv 16 Nov 2021.

## Builds on (most cited first, 9 of 9)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [Streaming automatic speech recognition with the transformer model](https://mlsanity.com/paper/2001.02674/) (2020)

## Built on by (most cited first, 9 of 9)

- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Automatically Auditing Large Language Models via Discrete Optimization](https://mlsanity.com/paper/2303.04381/) (2023)
- [GitHub Copilot AI pair programmer: Asset or Liability?](https://mlsanity.com/paper/2206.15331/) (2022)
- [Capturing Failures of Large Language Models via Human Cognitive Biases](https://mlsanity.com/paper/2202.12299/) (2022)
- [Large Language Models and Simple, Stupid Bugs](https://mlsanity.com/paper/2303.11455/) (2023)
- [Codex Hacks HackerRank: Memorization Issues and a Framework for Code Synthesis Evaluation](https://mlsanity.com/paper/2212.02684/) (2022)
- [A Survey on Pretrained Language Models for Neural Code Intelligence](https://mlsanity.com/paper/2212.10079/) (2022)
- [Neural Machine Translation for Code Generation](https://mlsanity.com/paper/2305.13504/) (2023)
- [CogNarr Ecosystem: Facilitating Group Cognition at Scale](https://mlsanity.com/paper/2407.18945/) (2024)

## Abstract (arXiv)

We solve university level probability and statistics questions by program synthesis using OpenAI's Codex, a Transformer trained on text and fine-tuned on code. We transform course problems from MIT's 18.05 Introduction to Probability and Statistics and Harvard's STAT110 Probability into programming tasks. We then execute the generated code to get a solution. Since these course questions are grounded in probability, we often aim to have Codex generate probabilistic programs that simulate a large number of probabilistic dependencies to compute its solution. Our approach requires prompt engineering to transform the question from its original form to an explicit, tractable form that results in a correct program and solution. To estimate the amount of work needed to translate an original question into its tractable form, we measure the similarity between original and transformed questions. Our work is the first to introduce a new dataset of university-level probability and statistics problems and solve these problems in a scalable fashion using the program synthesis capabilities of large language models.