DevIntent: How Much Does LLM-Generated Code Violate Developer Intent?
Not on the map. Why?
Abstract (arXiv)
Code generated by LLMs can violate a developer's implicit intentions when given an ambiguous prompt, yet standard benchmarks measure only whether code passes its stated test. We introduce the Intent Violation Rate (IVR) and a 49-problem pilot benchmark derived from HumanEval+. Each problem strips implicit constraints from a clarified prompt and encodes them as hidden constraint tests. IVR measures the fraction of LLM-generated solutions that pass the stated (visible) tests yet fail hidden constraint tests that capture unstated intent. Evaluating Claude Sonnet 4.6 and OpenAI GPT 4.1, we find both pass over 92% of stated tests yet violate intent in over half of problems (54.5% and 63.5%), following a systematic, bimodal pattern consistent across both models. Out findings indicate that pass rates overstate how well generated code reflects developer intent.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 7 of its 10 references that are in the map. First posted on arXiv 7 Aug 2026.
Builds on
7 of 10 references.
References · 7
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
- 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
- 2024Aligning Language Models to Explicitly Handle Ambiguitycited by 56
- 2023ClarifyGPT: Empowering LLM-based Code Generation with Intention Clarificationcited by 38
- 2026ClarEval: A Benchmark for Evaluating Clarification Skills of Code Agents under Ambiguous Instructionscited by 8
Built on by
Not cited yet.