mlsanityData to 7 Oct 2026

PPM: Automated Generation of Diverse Programming Problems for Benchmarking Code Generation Models

arXiv 2401.15545 ↗First posted 28 Jan 2024cs.SE cs.AI cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

In recent times, a plethora of Large Code Generation Models (LCGMs) have been proposed, showcasing significant potential in assisting developers with complex programming tasks. Benchmarking LCGMs necessitates the creation of a set of diverse programming problems, and each problem comprises the prompt (including the task description), canonical solution, and test inputs. The existing methods for constructing such a problem set can be categorized into two main types: manual methods and perturbation-based methods. However, manual methods demand high effort and lack scalability, while also risking data integrity due to LCGMs' potentially contaminated data collection, and perturbation-based approaches mainly generate semantically homogeneous problems with the same canonical solutions and introduce typos that can be easily auto-corrected by IDE, making them ineffective and unrealistic. In this work, we propose the idea of programming problem merging (PPM) and provide two implementation of this idea, we utilize our tool on two widely-used datasets and compare it against nine baseline methods using eight code generation models. The results demonstrate the effectiveness of our tool in generating more challenging, diverse, and natural programming problems, comparing to the baselines.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 32 of its 43 references that are in the map. First posted on arXiv 28 Jan 2024.

Builds on

32 of 43 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  7. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  8. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  9. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  10. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  11. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  12. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
20 more
  1. 2018code2vec: Learning Distributed Representations of Codecited by 249
  2. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  3. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  4. 2018code2seq: Generating Sequences from Structured Representations of Codecited by 177
  5. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  6. 2023CodeGen2: Lessons for Training LLMs on Programming and Natural Languagescited by 124
  7. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 100
  8. 2022Natural Attack for Pre-trained Models of Codecited by 72
  9. 2022Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examplescited by 65
  10. 2023On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilotcited by 64
  11. 2019Adversarial Examples for Models of Codecited by 61
  12. 2022NICGSlowDown: Evaluating the Efficiency Robustness of Neural Image Caption Generation Modelscited by 47
  13. 2020Semantic Robustness of Models of Source Codecited by 43
  14. 2023On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codexcited by 41
  15. 2023Pretraining on the Test Set Is All You Needcited by 33
  16. 2020Exploring Software Naturalness through Neural Language Modelscited by 32
  17. 2023Stealthy Backdoor Attack for Code Modelscited by 32
  18. 2021Adversarial Robustness of Deep Code Comment Generationcited by 22
  19. 2022Learning to Reverse DNNs from AI Programs Automaticallycited by 11
  20. 2022Poison Attack and Defense on Deep Source Code Processing Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 1.

7 papers.

  1. 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 106
  2. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  3. 2024FMint: Bridging Human Designed and Data Pretrained Models for Differential Equation Foundation Modelcited by 18
  4. 2025TreeDiff: AST-Guided Code Generation with Diffusion LLMscited by 11
  5. 2025REFINE: Enhancing Program Repair Agents through Context-Aware Patch Refinementcited by 6
  6. 2026ClarEval: A Benchmark for Evaluating Clarification Skills of Code Agents under Ambiguous Instructionscited by 5
  7. 2025AppForge: From Assistant to Independent Developer -- Are GPTs Ready for Software Development?cited by 4