mlsanityData to 7 Oct 2026

LEVER: Learning to Verify Language-to-Code Generation with Execution

arXiv 2302.08468 ↗First posted 16 Feb 2023cs.LG cs.CL cs.PLcited by 228

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

The advent of large language models trained on code (code LLMs) has led to significant progress in language-to-code generation. State-of-the-art approaches in this area combine LLM decoding with sample pruning and reranking using test cases or heuristics based on the execution results. However, it is challenging to obtain test cases for many real-world language-to-code applications, and heuristics cannot well capture the semantic features of the execution results, such as data type and value range, which often indicates the correctness of the program. In this work, we propose LEVER, a simple approach to improve language-to-code generation by learning to verify the generated programs with their execution results. Specifically, we train verifiers to determine whether a program sampled from the LLMs is correct or not based on the natural language input, the program itself and its execution results. The sampled programs are reranked by combining the verification score with the LLM generation probability, and marginalizing over programs with the same execution results. On four datasets across the domains of table QA, math QA and basic Python programming, LEVER consistently improves over the base code LLMs(4.6% to 10.9% with code-davinci-002) and achieves new state-of-the-art results on all of them.

Key facts. Cited by 228 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 49 of its 76 references that are in the map. First posted on arXiv 16 Feb 2023.

Builds on

49 of 76 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Competition-Level Code Generation with AlphaCode8 Feb

Other references · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  12. 2022Language Models (Mostly) Know What They Knowcited by 1,783
36 more
  1. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  2. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  3. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  4. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  5. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  6. 2022PAL: Program-aided Language Modelscited by 635
  7. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  8. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  9. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  10. 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 420
  11. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 402
  12. 2022CodeT: Code Generation with Generated Testscited by 369
  13. 2022Generating Sequences by Learning to Self-Correctcited by 271
  14. 2021TAPEX: Table Pre-training via Learning a Neural SQL Executorcited by 233
  15. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  16. 2022Binding Language Models in Symbolic Languagescited by 224
  17. 2016Language to Logical Form with Neural Attentioncited by 158
  18. 2022Natural Language to Code Translation with Executioncited by 122
  19. 2020Unit Test Case Generation with Transformers and Focal Contextcited by 118
  20. 2022Evaluating the Text-to-SQL Capabilities of Large Language Modelscited by 101
  21. 2018NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating Systemcited by 97
  22. 2017Abstract Syntax Networks for Code Generation and Semantic Parsingcited by 92
  23. 2022RASAT: Integrating Relational Structures into Pretrained Seq2Seq Model for Text-to-SQLcited by 85
  24. 2022Coder Reviewer Reranking for Code Generationcited by 83
  25. 2021Generate & Rank: A Multi-task Framework for Math Word Problemscited by 82
  26. 2020On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queriescited by 59
  27. 2018Robust Text-to-SQL Generation with Execution-Guided Decodingcited by 51
  28. 2019JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generationcited by 49
  29. 2017From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihoodcited by 41
  30. 2021Latent Execution for Neural Program Synthesiscited by 35
  31. 2022OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answeringcited by 29
  32. 2022TaCube: Pre-computing Data Cubes for Answering Numerical-Reasoning Questions over Tabular Datacited by 19
  33. 2017Weakly-supervised Semantic Parsing with Abstract Examplescited by 13
  34. 2015Multi-Document Summarization via Discriminative Summary Rerankingcited by 6
  35. 2018Multi-turn Dialogue Response Generation in an Adversarial Learning Frameworkcited by 6
  36. 2018Weakly-supervised Neural Semantic Parsing with a Generative Rankercited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 42; 2024: 92; 2025: 37; 2026: 57.

228 papers.

Sort:

By area:

  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  2. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  3. 2023Teaching Large Language Models to Self-Debugcited by 963
  4. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  5. 2024A Survey on Large Language Models for Code Generationcited by 646
  6. 2023Cognitive Architectures for Language Agentscited by 422
  7. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  8. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  9. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  10. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  11. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  12. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
38 more
  1. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
  2. 2024Many-Shot In-Context Learningcited by 203
  3. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  4. 2024Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understandingcited by 185
  5. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  6. 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 140
  7. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  8. 2024CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQLcited by 132
  9. 2023Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generationcited by 128
  10. 2024Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-stepcited by 123
  11. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 118
  12. 2024Why Tabular Foundation Models Should Be a Research Prioritycited by 103
  13. 2023A Survey of Text Watermarking in the Era of Large Language Modelscited by 102
  14. 2023SatLM: Satisfiability-Aided Language Models Using Declarative Promptingcited by 100
  15. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Modelscited by 95
  16. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  17. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinementcited by 89
  18. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  19. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  20. 2023Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaborationcited by 70
  21. 2024TroVE: Inducing Verifiable and Efficient Toolboxes for Solving Programmatic Taskscited by 70
  22. 2023Exploring Chain-of-Thought Style Prompting for Text-to-SQLcited by 69
  23. 2023MathChat: Converse to Tackle Challenging Math Problems with LLM Agentscited by 67
  24. 2024A Survey on Employing Large Language Models for Text-to-SQL Taskscited by 67
  25. 2024NExT: Teaching Large Language Models to Reason about Code Executioncited by 60
  26. 2023CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modulescited by 58
  27. 2024Efficiently Scaling LLM Reasoning with Certaindexcited by 56
  28. 2023FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answeringcited by 51
  29. 2024RSL-SQL: Robust Schema Linking in Text-to-SQL Generationcited by 51
  30. 2024Calibration and Correctness of Language Models for Codecited by 49
  31. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  32. 2023Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistencycited by 46
  33. 2024When is Tree Search Useful for LLM Planning? It Depends on the Discriminatorcited by 46
  34. 2024From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debuggingcited by 46
  35. 2023Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategiescited by 44
  36. 2024Reasoning Runtime Behavior of a Program with LLM: How Far Are We?cited by 44
  37. 2023SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended)cited by 42
  38. 2023ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifierscited by 41

50 most cited of 228 All 228 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning to Solve and Verify: A Self-Play Framework for Code and Test Generation
  2. 2026Are you Synthesizing or Recalling? Evaluating LLMs on Algorithmic Code Retrieval
  3. 2025Prompt engineering and framework: implementation to increase code reliability based guideline for LLMs
  4. 2025Can LLMs Generate Reliable Test Case Generators? A Study on Competition-Level Programming Problems
  5. 2026CodeEval: A pedagogical approach for targeted evaluation of code-trained Large Language Models
  6. 2025HardTests: Synthesizing High-Quality Test Cases for LLM Coding
  7. 2023CAT-LM: Training Language Models on Aligned Code And Tests
  8. 2024CodeLutra: Boosting LLM Code Generation via Preference-Guided Refinement