mlsanityData to 14 Oct 2026

More Than a Score: Probing the Impact of Prompt Specificity on LLM Code Generation

arXiv 2508.03678 ↗First posted 5 Aug 2025cs.CL cs.LG cs.PLcited by 4

Not on the map. Why?

Abstract (arXiv)

State-of-the-art Large Language Models (LLMs) achieve high pass@1 on general benchmarks like HumanEval but underperform on specialized suites such as ParEval. Is this due to LLMs missing domain knowledge or insufficient prompt detail is given? To answer this, we introduce PartialOrderEval, which augments any code generation benchmark with a partial order of prompts from minimal to maximally detailed. Applying it to HumanEval and both serial and OpenMP subsets of ParEval, we measure how pass@1 scales with prompt specificity. Our experiments with Llama-3.x and Qwen2.5-Coder demonstrate varying degrees of prompt sensitivity across different tasks, and a qualitative analysis highlights explicit I/O specifications, edge-case handling, and stepwise breakdowns as the key drivers of prompt detail improvement.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 23 of its 64 references that are in the map. First posted on arXiv 5 Aug 2025.

Builds on

23 of 64 references.

References · 23

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  6. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
  7. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
  8. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 226
  9. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 136
  10. 2024HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalizationcited by 86
  11. 2023How Novices Use LLM-Based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environmentcited by 56
  12. 2024The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmerscited by 48
11 more
  1. 2023Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMscited by 45
  2. 2022MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languagescited by 41
  3. 2024Can Large Language Models Write Parallel Code?cited by 36
  4. 2022Robustness of Learning from Task Instructionscited by 35
  5. 2023Measuring The Impact Of Programming Language Distributioncited by 32
  6. 2023StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Codecited by 32
  7. 2024mHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code Generationcited by 32
  8. 2024How Beginning Programmers and Code LLMs (Mis)read Each Othercited by 31
  9. 2024PromptSet: A Programmer's Prompting Datasetcited by 9
  10. 2024Substance Beats Style: Why Beginning Students Fail to Code with LLMscited by 5
  11. 2024NoviCode: Generating Programs from Natural Language Utterances by Novicescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Adding New Capability in Existing Scientific Application with LLM Assistancecited by 1
  2. 2026PROMPT2BOX:Improving LLM Weakness Discovery and Specificity Estimation by Uncovering Entailment Structure among Promptscited by 0
  3. 2026Beyond the Payload: How User Invocation Shapes Coding Agent Vulnerability to Repository Poisoningcited by 0
  4. 2026Implicit Manipulation for Skill Selection in LLM Agents with Semantic Matchingcited by 0