mlsanityData to 7 Oct 2026

CodeT: Code Generation with Generated Tests

arXiv 2207.10397 ↗First posted 21 Jul 2022cs.CL cs.AI cs.PLcited by 369

Not on the map. Why?

Cited by 63 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The task of generating code solutions for a given programming problem can benefit from the use of pre-trained language models such as Codex, which can produce multiple diverse samples. However, a major challenge for this task is to select the most appropriate solution from the multiple samples generated by the pre-trained language models. A natural way to evaluate the quality and correctness of a code solution is to run it against a set of test cases, but the manual creation of such test cases is often costly and time-consuming. In this paper, we propose a novel method, CodeT, that leverages the same pre-trained language models to automatically generate test cases for the code samples, thus reducing the human effort and increasing the coverage of the test scenarios. CodeT then executes the code samples using the generated test cases, and performs a dual execution agreement, which considers both the consistency of the outputs against the generated test cases and the agreement of the outputs with other code samples. We conduct comprehensive experiments on four benchmarks, HumanEval, MBPP, APPS and CodeContests, using five different pre-trained language models with varying sizes and capabilities. Our results show that CodeT can significantly improve the performance of code solution selection over previous methods, achieving remarkable and consistent gains across different models and benchmarks. For instance, CodeT improves the pass@1 metric on HumanEval to 65.8%, which represents an absolute improvement of 18.8% over the code-davinci-002 model, and an absolute improvement of more than 20% over the previous state-of-the-art results.

Key facts. Cited by 369 papers in the map (data through 7 Oct 2026). Builds on 26 of its 41 references that are in the map. First posted on arXiv 21 Jul 2022.

Builds on

26 of 41 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  9. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  10. 2021Measuring Coding Challenge Competence With APPScited by 879
  11. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  12. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
14 more
  1. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  2. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  3. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  4. 2022Automatic Generation of Programming Exercises and Code Explanations using Large Language Modelscited by 124
  5. 2022Natural Language to Code Translation with Executioncited by 122
  6. 2020Unit Test Case Generation with Transformers and Focal Contextcited by 118
  7. 2021A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Levelcited by 96
  8. 2019Write, Execute, Assess: Program Synthesis with a REPLcited by 92
  9. 2021Generate & Rank: A Multi-task Framework for Math Word Problemscited by 82
  10. 2021Leveraging Automated Unit Tests for Unsupervised Code Translationcited by 82
  11. 2022Pynguin: Automated Unit Test Generation for Pythoncited by 49
  12. 2022Fault-Aware Neural Code Rankerscited by 33
  13. 2017BiRank: Towards Ranking on Bipartite Graphscited by 15
  14. 2019Many Independent Objective (MIO) Algorithm for Test Suite Generationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 68; 2024: 98; 2025: 89; 2026: 105.

369 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  5. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  6. 2023Teaching Large Language Models to Self-Debugcited by 963
  7. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  8. 2024A Survey on Large Language Models for Code Generationcited by 646
  9. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  10. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  11. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  12. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
38 more
  1. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  2. 2026Reinforcement Learning via Self-Distillationcited by 285
  3. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  4. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 247
  5. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 246
  6. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  7. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  8. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  9. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  10. 2022Binding Language Models in Symbolic Languagescited by 224
  11. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  12. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 193
  13. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 191
  14. 2023Planning with Large Language Models for Code Generationcited by 179
  15. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  16. 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solvingcited by 177
  17. 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 140
  18. 2023Structured Chain-of-Thought Prompting for Code Generationcited by 136
  19. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  20. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  21. 2024SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agentscited by 129
  22. 2023Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generationcited by 128
  23. 2024Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-stepcited by 123
  24. 2024FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Makingcited by 112
  25. 2023Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMscited by 111
  26. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 110
  27. 2023Learning Performance-Improving Code Editscited by 103
  28. 2025S*: Test Time Scaling for Code Generationcited by 95
  29. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 89
  30. 2025ACECODER: Acing Coder RL via Automated Test-Case Synthesiscited by 89
  31. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  32. 2023LLMs cannot find reasoning errors, but can correct them given the error locationcited by 85
  33. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  34. 2023Is ChatGPT the Ultimate Programming Assistant -- How far is it?cited by 72
  35. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  36. 2023PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedbackcited by 69
  37. 2025Paper2Code: Automating Code Generation from Scientific Papers in Machine Learningcited by 67
  38. 2024Aligning LLMs with Individual Preferences via Interactioncited by 63

50 most cited of 369 All 369 (JSON)