mlsanityData to 14 Oct 2026

DOCE: Finding the Sweet Spot for Execution-Based Code Generation

arXiv 2408.13745 ↗First posted 25 Aug 2024cs.CL cs.AI cs.PLcited by 4

Not on the map. Why?

Abstract (arXiv)

Recently, a diverse set of decoding and reranking procedures have been shown effective for LLM-based code generation. However, a comprehensive framework that links and experimentally compares these methods is missing. We address this by proposing Decoding Objectives for Code Execution, a comprehensive framework that includes candidate generation, n-best reranking, minimum Bayes risk (MBR) decoding, and self-debugging as the core components. We then study the contributions of these components through execution-based evaluation metrics. Our findings highlight the importance of execution-based methods and the difference gap between execution-based and execution-free methods. Furthermore, we assess the impact of filtering based on trial unit tests, a simple and effective strategy that has been often overlooked in prior works. We also propose self-debugging on multiple candidates, obtaining state-of-the-art performance on reranking for code generation. We expect our framework to provide a solid guideline for future research on code generation.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 25 Aug 2024.

Builds on

41 of 55 references.

References · 41

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  9. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  10. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  11. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  12. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
29 more
  1. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  2. 2023Teaching Large Language Models to Self-Debugcited by 988
  3. 2021Measuring Coding Challenge Competence With APPScited by 902
  4. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  5. 2023StarCoder: may the source be with you!cited by 799
  6. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  7. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  8. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  9. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  10. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  11. 2022CodeT: Code Generation with Generated Testscited by 384
  12. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  13. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 239
  14. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 197
  15. 2019Facebook FAIR's WMT19 News Translation Task Submissioncited by 179
  16. 2020Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translationcited by 138
  17. 2022Natural Language to Code Translation with Executioncited by 124
  18. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  19. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 102
  20. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 91
  21. 2022Coder Reviewer Reranking for Code Generationcited by 84
  22. 2021Generate & Rank: A Multi-task Framework for Math Word Problemscited by 82
  23. 2023CodeScore: Evaluating Code Generation by Learning Code Executioncited by 67
  24. 2021Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translationcited by 65
  25. 2023ICE-Score: Instructing Large Language Models to Evaluate Codecited by 52
  26. 2022Quality-Aware Decoding for Neural Machine Translationcited by 49
  27. 2023Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistencycited by 46
  28. 2020Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Modelscited by 44
  29. 2023Functional Overlap Reranking for Neural Code Generationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 3.

4 papers.

  1. 2024Efficiently Scaling LLM Reasoning with Certaindexcited by 58
  2. 2026ACES: Who Tests the Tests? Leave-One-Out AUC Consistency for Code Generationcited by 2
  3. 2026Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Modelscited by 0
  4. 2026Characterizing Overconfident Failure in LLM-Based Code Generationcited by 0