mlsanityData to 7 Oct 2026

Coder Reviewer Reranking for Code Generation

arXiv 2211.16490 ↗First posted 29 Nov 2022cs.LG cs.CL cs.PLcited by 83

Not on the map. Why?

Abstract (arXiv)

Sampling diverse programs from a code language model and reranking with model likelihood is a popular method for code generation but it is prone to preferring degenerate solutions. Inspired by collaborative programming, we propose Coder-Reviewer reranking. We augment Coder language models from past work, which generate programs given language instructions, with Reviewer models, which evaluate the likelihood of the instruction given the generated programs. We perform an extensive study across six datasets with eight models from three model families. Experimental results show that Coder-Reviewer reranking leads to consistent and significant improvement (up to 17% absolute accuracy gain) over reranking with the Coder model only. When combined with executability filtering, Coder-Reviewer reranking can often outperform the minimum Bayes risk method. Coder-Reviewer reranking is easy to implement by prompting, can generalize to different programming languages, and works well with off-the-shelf hyperparameters.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Builds on 23 of its 40 references that are in the map. First posted on arXiv 29 Nov 2022.

Builds on

23 of 40 references.

References · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  6. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  7. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  8. 2021Measuring Coding Challenge Competence With APPScited by 879
  9. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  10. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  11. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  12. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
11 more
  1. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 164
  2. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  3. 2022Natural Language to Code Translation with Executioncited by 122
  4. 2019On NMT Search Errors and Model Errors: Cat Got Your Tongue?cited by 114
  5. 2018Mapping Language to Code in Programmatic Contextcited by 104
  6. 2018NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating Systemcited by 97
  7. 2020Graph-based, Self-Supervised Program Repair from Diagnostic Feedbackcited by 93
  8. 2016Latent Predictor Networks for Code Generationcited by 73
  9. 2022Fault-Aware Neural Code Rankerscited by 33
  10. 2022Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learnerscited by 22
  11. 2022Non-Programmers Can Label Programs Indirectly via Active Examples: A Case Study with Text-to-SQLcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 24; 2024: 35; 2025: 11; 2026: 11.

83 papers.

Sort:

By area:

  1. 2023Teaching Large Language Models to Self-Debugcited by 963
  2. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  3. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  4. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  5. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  6. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  7. 2023Self-Evaluation Guided Beam Search for Reasoningcited by 195
  8. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 191
  9. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  10. 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 140
  11. 2024Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-stepcited by 123
  12. 2023Hypothesis Search: Inductive Reasoning with Language Modelscited by 119
38 more
  1. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 118
  2. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 110
  3. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 86
  4. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  5. 2024SpecRover: Code Intent Extraction via LLMscited by 71
  6. 2022Parsel: Algorithmic Reasoning with Language Models by Composing Decompositionscited by 62
  7. 2023Clover: Closed-Loop Verifiable Code Generationcited by 62
  8. 2023CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modulescited by 58
  9. 2024Calibration and Correctness of Language Models for Codecited by 49
  10. 2023Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistencycited by 46
  11. 2024Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedbackcited by 45
  12. 2023ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifierscited by 41
  13. 2024CodeTree: Agent-guided Tree Search for Code Generation with Large Language Modelscited by 32
  14. 2023CodeCoT: Tackling Code Syntax Errors in CoT Reasoning for Code Generationcited by 29
  15. 2023L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Modelscited by 22
  16. 2024A Pair Programming Framework for Code Generation via Multi-Plan Exploration and Feedback-Driven Refinementcited by 21
  17. 2024The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations?cited by 20
  18. 2025Reliable Text-to-SQL with Adaptive Abstentioncited by 19
  19. 2023INTERVENOR: Prompting the Coding Ability of Large Language Models with the Interactive Chain of Repaircited by 18
  20. 2024Enhancing Code Generation Performance of Smaller Models by Distilling the Reasoning Ability of LLMscited by 17
  21. 2024Leveraging Environment Interaction for Automated PDDL Translation and Planning with Large Language Modelscited by 17
  22. 2024Path-Consistency with Prefix Enhancement for Efficient Inference in LLMscited by 17
  23. 2024Syzygy: Dual Code-Test C to (safe) Rust Translation using LLMs and Dynamic Analysiscited by 16
  24. 2024Leveraging Print Debugging to Improve Code Generation in Large Language Modelscited by 15
  25. 2024VerMCTS: Synthesizing Multi-Step Programs using a Verifier, a Large Language Model, and Tree Searchcited by 13
  26. 2024Improving Minimum Bayes Risk Decoding with Multi-Promptcited by 13
  27. 2023Explain-then-Translate: An Analysis on Improving Program Translation with Self-generated Explanationscited by 12
  28. 2023From Words to Code: Harnessing Data for Program Synthesis from Natural Languagecited by 11
  29. 2023R-U-SURE? Uncertainty-Aware Code Suggestions By Maximizing Utility Across Random User Intentscited by 10
  30. 2023Language Models are Bounded Pragmatic Speakers: Understanding RLHF from a Bayesian Cognitive Modeling Perspectivecited by 10
  31. 2023Lightweight reranking for language model generationscited by 10
  32. 2024Compositional API Recommendation for Library-Oriented Code Generationcited by 10
  33. 2024Preference-Guided Refactored Tuning for Retrieval Augmented Code Generationcited by 10
  34. 2023Functional Overlap Reranking for Neural Code Generationcited by 9
  35. 2024Top Pass: Improve Code Generation by Pass@k-Maximized Code Rankingcited by 9
  36. 2024B4: Towards Optimal Assessment of Plausible Code Solutions with Plausible Testscited by 9
  37. 2024CoopetitiveV: Leveraging LLM-powered Coopetitive Multi-Agent Prompting for High-quality Verilog Generationcited by 9
  38. 2024EPI-SQL: Enhancing Text-to-SQL Translation with Error-Prevention Instructionscited by 8

50 most cited of 83 All 83 (JSON)