mlsanityData to 14 Oct 2026

Natural Language to Code Translation with Execution

arXiv 2204.11454 ↗First posted 25 Apr 2022cs.CL cs.SEcited by 124

Not on the map. Why?

Cited by 24 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generative models of code, pretrained on large corpora of programs, have shown great success in translating natural language to code (Chen et al., 2021; Austin et al., 2021; Li et al., 2022, inter alia). While these models do not explicitly incorporate program semantics (i.e., execution results) during training, they are able to generate correct solutions for many problems. However, choosing a single correct program from a generated set for each problem remains challenging. In this work, we introduce execution result--based minimum Bayes risk decoding (MBR-EXEC) for program selection and show that it improves the few-shot performance of pretrained code models on natural-language-to-code tasks. We select output programs from a generated candidate set by marginalizing over program implementations that share the same semantics. Because exact equivalence is intractable, we execute each program on a small number of test inputs to approximate semantic equivalence. Across datasets, execution or simulated execution significantly outperforms the methods that do not involve program semantics. We find that MBR-EXEC consistently improves over all execution-unaware selection methods, suggesting it as an effective approach for natural language to code translation. We open-source our code at github.com/facebookresearch/mbr-exec and data at dl.fbaipublicfiles.com/mbr-exec/mbr-exec-release.zip

Key facts. Cited by 124 papers in the map (data through 14 Oct 2026). Builds on 36 of its 58 references that are in the map. First posted on arXiv 25 Apr 2022.

Builds on

36 of 58 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2021Program Synthesis with Large Language Modelscited by 3,637
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  10. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  11. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  12. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
24 more
  1. 2019How Can We Know What Language Models Know?cited by 959
  2. 2021Measuring Coding Challenge Competence With APPScited by 902
  3. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  4. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  5. 2019Neural Text Generation with Unlikelihood Trainingcited by 503
  6. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  7. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 411
  8. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  9. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  10. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
  11. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  12. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 165
  13. 2020Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translationcited by 138
  14. 2020Semantic Evaluation for Text-to-SQL with Distilled Test Suitescited by 126
  15. 2018Coarse-to-Fine Decoding for Neural Semantic Parsingcited by 110
  16. 2018NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating Systemcited by 98
  17. 2017Abstract Syntax Networks for Code Generation and Semantic Parsingcited by 92
  18. 2016Latent Predictor Networks for Code Generationcited by 73
  19. 2021DOBF: A Deobfuscation Pre-Training Objective for Programming Languagescited by 68
  20. 2017A parallel corpus of Python functions and documentation strings for automated code documentation and code generationcited by 57
  21. 2019Visually Grounded Neural Syntax Acquisitioncited by 48
  22. 2020Efficient Second-Order TreeCRF for Neural Dependency Parsingcited by 48
  23. 2020Incorporating External Knowledge through Pre-training for Natural Language to Code Generationcited by 36
  24. 2018Learning to Map Context-Dependent Sentences to Executable Formal Queriescited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 11; 2023: 35; 2024: 39; 2025: 19; 2026: 20.

124 papers.

Sort:

By area:

  1. 2023Teaching Large Language Models to Self-Debugcited by 988
  2. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  3. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  4. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  5. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  6. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  7. 2023Augmented Language Models: a Surveycited by 403
  8. 2022CodeT: Code Generation with Generated Testscited by 384
  9. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 323
  10. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  11. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 251
  12. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 240
38 more
  1. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 239
  2. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 238
  3. 2022Binding Language Models in Symbolic Languagescited by 228
  4. 2023Universal Self-Consistency for Large Language Model Generationcited by 203
  5. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 197
  6. 2025Dynamic Cheatsheet: Test-Time Learning with Adaptive Memorycited by 163
  7. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 160
  8. 2022Multi-lingual Evaluation of Code Generation Modelscited by 159
  9. 2022DocPrompting: Generating Code by Retrieving the Docscited by 153
  10. 2023Compositional Exemplars for In-context Learningcited by 144
  11. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  12. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 131
  13. 2024Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-stepcited by 127
  14. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 120
  15. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 111
  16. 2022Compositional Semantic Parsing with Large Language Modelscited by 95
  17. 2022Coder Reviewer Reranking for Code Generationcited by 84
  18. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 79
  19. 2024TroVE: Inducing Verifiable and Efficient Toolboxes for Solving Programmatic Taskscited by 75
  20. 2023Clover: Closed-Loop Verifiable Code Generationcited by 67
  21. 2023CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modulescited by 58
  22. 2022Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decodingcited by 54
  23. 2023It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Riskcited by 51
  24. 2024Self-Consistency Preference Optimizationcited by 50
  25. 2023Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistencycited by 46
  26. 2024Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedbackcited by 46
  27. 2025Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlocited by 46
  28. 2023SelfEvolve: A Code Evolution Framework via Large Language Modelscited by 43
  29. 2024The Larger the Better? Improved LLM Code-Generation via Budget Reallocationcited by 43
  30. 2023ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifierscited by 41
  31. 2023Faster Minimum Bayes Risk Decoding with Confidence-based Pruningcited by 34
  32. 2024CodeTree: Agent-guided Tree Search for Code Generation with Large Language Modelscited by 34
  33. 2025Uncertainty-Guided Chain-of-Thought for Code Generation with LLMscited by 32
  34. 2023Learning Deep Semantics for Test Completioncited by 31
  35. 2024InternalInspector I2: Robust Confidence Estimation in LLMs through Internal Statescited by 30
  36. 2023CodeCoT: Tackling Code Syntax Errors in CoT Reasoning for Code Generationcited by 29
  37. 2024What Makes Large Language Models Reason in (Multi-Turn) Code Generation?cited by 29
  38. 2023L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Modelscited by 22

50 most cited of 124 All 124 (JSON)