mlsanityData to 7 Oct 2026

Fault-Aware Neural Code Rankers

arXiv 2206.03865 ↗First posted 4 Jun 2022cs.PL cs.AI cs.SEcited by 33

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have demonstrated an impressive ability to generate code for various programming tasks. In many instances, LLMs can generate a correct program for a task when given numerous trials. Consequently, a recent trend is to do large scale sampling of programs using a model and then filtering/ranking the programs based on the program execution on a small number of known unit tests to select one candidate solution. However, these approaches assume that the unit tests are given and assume the ability to safely execute the generated programs (which can do arbitrary dangerous operations such as file manipulations). Both of the above assumptions are impractical in real-world software development. In this paper, we propose CodeRanker, a neural ranker that can predict the correctness of a sampled program without executing it. Our CodeRanker is fault-aware i.e., it is trained to predict different kinds of execution information such as predicting the exact compile/runtime error type (e.g., an IndexError or a TypeError). We show that CodeRanker can significantly increase the pass@1 accuracy of various code generation models (including Codex, GPT-Neo, GPT-J) on APPS, HumanEval and MBPP datasets.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 29 of its 50 references that are in the map. First posted on arXiv 4 Jun 2022.

Builds on

29 of 50 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  9. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  10. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  11. 2021Measuring Coding Challenge Competence With APPScited by 879
  12. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
17 more
  1. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  2. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  3. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  4. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  5. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  6. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 266
  7. 2019SPoC: Search-based Pseudocode to Codecited by 176
  8. 2020Deep Learning based Vulnerability Detection: Are We There Yet?cited by 142
  9. 2018Tree-to-tree Neural Networks for Program Translationcited by 93
  10. 2019Write, Execute, Assess: Program Synthesis with a REPLcited by 92
  11. 2021Generate & Rank: A Multi-task Framework for Math Word Problemscited by 82
  12. 2017A parallel corpus of Python functions and documentation strings for automated code documentation and code generationcited by 56
  13. 2019When Deep Learning Met Code Searchcited by 45
  14. 2021Latent Execution for Neural Program Synthesiscited by 35
  15. 2022Training and Evaluating a Jupyter Notebook Data Science Assistantcited by 31
  16. 2021Programming Puzzlescited by 25
  17. 2021Neural Program Generation Modulo Static Analysiscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 13; 2024: 6; 2025: 4; 2026: 4.

33 papers.

Sort:

By area:

  1. 2022CodeT: Code Generation with Generated Testscited by 369
  2. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 191
  3. 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 140
  4. 2023Structured Chain-of-Thought Prompting for Code Generationcited by 136
  5. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 110
  6. 2022Coder Reviewer Reranking for Code Generationcited by 83
  7. 2022Repair Is Nearly Generation: Multilingual Program Repair with LLMscited by 68
  8. 2023Clover: Closed-Loop Verifiable Code Generationcited by 62
  9. 2023CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modulescited by 58
  10. 2024Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategiescited by 44
  11. 2023Large Language Model-Aware In-Context Learning for Code Generationcited by 42
  12. 2023ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifierscited by 41
21 more
  1. 2023CodeCoT: Tackling Code Syntax Errors in CoT Reasoning for Code Generationcited by 29
  2. 2024Is Programming by Example solved by LLMs?cited by 27
  3. 2024Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarkingcited by 26
  4. 2023AceCoder: Utilizing Existing Code to Enhance Code Generationcited by 25
  5. 2023LeTI: Learning to Generate from Textual Interactionscited by 25
  6. 2022ContraCLM: Contrastive Learning For Causal Language Modelcited by 18
  7. 2023FRANC: A Lightweight Framework for High-Quality Code Generationcited by 14
  8. 2022Toward Trustworthy Neural Program Synthesiscited by 11
  9. 2023Comparing Software Developers with ChatGPT: An Empirical Investigationcited by 11
  10. 2024Top Pass: Improve Code Generation by Pass@k-Maximized Code Rankingcited by 9
  11. 2024Data Analysis in the Era of Generative AIcited by 6
  12. 2025On LLMs' Internal Representation of Code Correctnesscited by 5
  13. 2026Scaling Agentic Verifier for Competitive Codingcited by 5
  14. 2025RefleXGen:The unexamined code is not worth usingcited by 4
  15. 2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 4
  16. 2022Diverse Title Generation for Stack Overflow Posts with Multiple Sampling Enhanced Transformercited by 2
  17. 2023Coarse-Tuning Models of Code with Reinforcement Learning Feedbackcited by 2
  18. 2024Self-Explained Keywords Empower Large Language Models for Code Generationcited by 2
  19. 2025Reinforcement Learning-Guided Chain-of-Draft for Token-Efficient Code Generationcited by 1
  20. 2026MIST-RL: Mutation-based Incremental Suite Testing via Reinforcement Learningcited by 0
  21. 2026XSearch: Explainable Code Search via Concept-to-Code Alignmentcited by 0