mlsanityData to 14 Oct 2026

Toward Trustworthy Neural Program Synthesis

arXiv 2210.00848 ↗First posted 29 Sep 2022cs.SE cs.AI cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

We develop an approach to estimate the probability that a program sampled from a large language model is correct. Given a natural language description of a programming problem, our method samples both candidate programs as well as candidate predicates specifying how the program should behave. This allows learning a model that forms a well-calibrated probabilistic prediction of program correctness. Our system also infers which predicates are useful to explain the behavior of the generated code, and humans preferred these in a human study over raw language model outputs. Our method is simple, easy to implement, and maintains state of the art generation accuracy results.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 16 of its 32 references that are in the map. First posted on arXiv 29 Sep 2022.

Builds on

16 of 32 references.

References · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  9. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  10. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  11. 2022CodeT: Code Generation with Generated Testscited by 384
  12. 2022Natural Language to Code Translation with Executioncited by 124
4 more
  1. 2016A Survey of Symbolic Execution Techniquescited by 36
  2. 2021Communicating Natural Programs to Humans and Machinescited by 35
  3. 2022Fault-Aware Neural Code Rankerscited by 33
  4. 2020Program Synthesis with Pragmatic Communicationcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 2.

11 papers.

  1. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  2. 2024Calibration and Correctness of Language Models for Codecited by 49
  3. 2024SymbolicAI: A framework for logic-based approaches combining generative models and solverscited by 16
  4. 2024Data Analysis in the Era of Generative AIcited by 6
  5. 2025Agint: Agentic Graph Compilation for Software Engineering Agentscited by 4
  6. 2025Synthesizing High-Quality Programming Tasks with LLM-based Expert and Student Agentscited by 3
  7. 2025Effective LLM Code Refinement via Property-Oriented and Structurally Minimal Feedbackcited by 2
  8. 2025MultiMat: Multimodal Program Synthesis for Procedural Materials using Large Multimodal Modelscited by 1
  9. 2026Teaching Code LLMs to Reason with Intermediate Formal Specificationscited by 1
  10. 2025Natural Building Blocks for Structured World Models: Theory, Evidence, and Scalingcited by 0
  11. 2026An Empirical Study of Security Calibration in Large Language Models for Codecited by 0