# Toward Trustworthy Neural Program Synthesis

arXiv: https://arxiv.org/abs/2210.00848 · HTML: https://mlsanity.com/paper/2210.00848/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 29 Sep 2022.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [CodeT: Code Generation with Generated Tests](https://mlsanity.com/paper/2207.10397/) (2022)
- [Natural Language to Code Translation with Execution](https://mlsanity.com/paper/2204.11454/) (2022)
- [A Survey of Symbolic Execution Techniques](https://mlsanity.com/paper/1610.00502/) (2016)
- [Communicating Natural Programs to Humans and Machines](https://mlsanity.com/paper/2106.07824/) (2021)
- [Fault-Aware Neural Code Rankers](https://mlsanity.com/paper/2206.03865/) (2022)
- [Program Synthesis with Pragmatic Communication](https://mlsanity.com/paper/2007.05060/) (2020)

## Built on by (most cited first, 11 of 11)

- [CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution](https://mlsanity.com/paper/2401.03065/) (2024)
- [Calibration and Correctness of Language Models for Code](https://mlsanity.com/paper/2402.02047/) (2024)
- [SymbolicAI: A framework for logic-based approaches combining generative models and solvers](https://mlsanity.com/paper/2402.00854/) (2024)
- [Data Analysis in the Era of Generative AI](https://mlsanity.com/paper/2409.18475/) (2024)
- [Agint: Agentic Graph Compilation for Software Engineering Agents](https://mlsanity.com/paper/2511.19635/) (2025)
- [Synthesizing High-Quality Programming Tasks with LLM-based Expert and Student Agents](https://mlsanity.com/paper/2504.07655/) (2025)
- [Effective LLM Code Refinement via Property-Oriented and Structurally Minimal Feedback](https://mlsanity.com/paper/2506.18315/) (2025)
- [MultiMat: Multimodal Program Synthesis for Procedural Materials using Large Multimodal Models](https://mlsanity.com/paper/2509.22151/) (2025)
- [Teaching Code LLMs to Reason with Intermediate Formal Specifications](https://mlsanity.com/paper/2607.04232/) (2026)
- [Natural Building Blocks for Structured World Models: Theory, Evidence, and Scaling](https://mlsanity.com/paper/2511.02091/) (2025)
- [An Empirical Study of Security Calibration in Large Language Models for Code](https://mlsanity.com/paper/2606.31159/) (2026)

## Abstract (arXiv)

We develop an approach to estimate the probability that a program sampled from a large language model is correct. Given a natural language description of a programming problem, our method samples both candidate programs as well as candidate predicates specifying how the program should behave. This allows learning a model that forms a well-calibrated probabilistic prediction of program correctness. Our system also infers which predicates are useful to explain the behavior of the generated code, and humans preferred these in a human study over raw language model outputs. Our method is simple, easy to implement, and maintains state of the art generation accuracy results.