mlsanityData to 14 Oct 2026

Defective Task Descriptions in LLM-Based Code Generation: Detection and Analysis

arXiv 2604.24703 ↗First posted 27 Apr 2026cs.SE cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Large language models are widely used for code generation, yet they rely on an implicit assumption that the task descriptions are sufficiently detailed and well-formed. However, in practice, users may provide defective descriptions, which can have a strong effect on code correctness. To address this issue, we develop SpecValidator, a lightweight classifier based on a small model that has been parameter-efficiently finetuned, to automatically detect task description defects. We evaluate SpecValidator on three types of defects, Lexical Vagueness, Under-Specification and Syntax-Formatting on 3 benchmarks with task descriptions of varying structure and complexity. Our results show that SpecValidator achieves defect detection of F1 = 0.804 and MCC = 0.745, significantly outperforming GPT-5-mini (F1 = 0.469 and MCC = 0.281) and Claude Sonnet 4 (F1 = 0.518 and MCC = 0.359). Perhaps more importantly, our analysis indicates that SpecValidator can generalize to unseen issues and detect unknown Under-Specification defects in the original (real) descriptions of the benchmarks used. Our results also show that the robustness of LLMs in task description defects depends primarily on the type of defect and the characteristics of the task description, rather than the capacity of the model, with Under-Specification defects being the most severe. We further found that benchmarks with richer contextual grounding, such as LiveCodeBench, exhibit substantially greater resilience, highlighting the importance of structured task descriptions for reliable LLM-based code generation.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 27 of its 44 references that are in the map. First posted on arXiv 27 Apr 2026.

Builds on

27 of 44 references.

References · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  7. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  8. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  9. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  10. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  11. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  12. 2023Teaching Large Language Models to Self-Debugcited by 988
15 more
  1. 2024A Survey on Large Language Models for Code Generationcited by 658
  2. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  3. 2023The Impact of AI on Developer Productivity: Evidence from GitHub Copilotcited by 274
  4. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 226
  5. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 103
  6. 2024Quantifying Contamination in Evaluating Code Generation Capabilities of Language Modelscited by 65
  7. 2023On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilotcited by 64
  8. 2024On Leakage of Code Generation Evaluation Datasetscited by 32
  9. 2024HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agentcited by 24
  10. 2024NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variationscited by 20
  11. 2023Fixing Large Language Models' Specification Misunderstanding for Better Code Generationcited by 17
  12. 2025When Prompts Go Wrong: Evaluating Code Model Robustness to Ambiguous, Contradictory, and Incomplete Task Descriptionscited by 11
  13. 2025A Multi-Language Perspective on the Robustness of LLM Code Generationcited by 6
  14. 2025Automated Repair of Ambiguous Problem Descriptions for LLM-Based Code Generationcited by 5
  15. 2025On the Impact of Requirements Smells in Prompts: The Case of Automated Traceabilitycited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Agent System Operations: Categorization, Challenges, and Future Directionscited by 0
  2. 2026AssumptionMiner: Extracting, Tracing, and Revising Implicit Assumptions in LLM Code Generationcited by 0