mlsanityData to 14 Oct 2026

LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation

arXiv 2404.10100 ↗First posted 15 Apr 2024cs.SEcited by 49

Not on the map. Why?

Cited by 20 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have shown great potential in automating significant aspects of coding by producing natural code from informal natural language (NL) intent. However, given NL is informal, it does not lend easily to checking that the generated code correctly satisfies the user intent. In this paper, we propose a novel interactive workflow TiCoder for guided intent clarification (i.e., partial formalization) through tests to support the generation of more accurate code suggestions. Through a mixed methods user study with 15 programmers, we present an empirical evaluation of the effectiveness of the workflow to improve code generation accuracy. We find that participants using the proposed workflow are significantly more likely to correctly evaluate AI generated code, and report significantly less task-induced cognitive load. Furthermore, we test the potential of the workflow at scale with four different state-of-the-art LLMs on two python datasets, using an idealized proxy for a user feedback. We observe an average absolute improvement of 45.97% in the pass@1 code generation accuracy for both datasets and across all LLMs within 5 user interactions, in addition to the automatic generation of accompanying unit tests.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Apr 2024.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 23; 2026: 19.

49 papers.

Sort:

By area:

  1. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 86
  2. 2024SpecRover: Code Intent Extraction via LLMscited by 72
  3. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 42
  4. 2025Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineeringcited by 38
  5. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 28
  6. 2024HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agentcited by 24
  7. 2025debug-gym: A Text-Based Environment for Interactive Debuggingcited by 19
  8. 2025Large Language Models (LLMs) for Source Code Analysis: applications, models and datasetscited by 16
  9. 2024APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Servingcited by 15
  10. 2025Tool-integrated Reinforcement Learning for Repo Deep Searchcited by 15
  11. 2025Unified attacks to large language model watermarks: spoofing and scrubbing in unauthorized knowledge distillationcited by 11
  12. 2024PLUM: Improving Code LMs with Execution-Guided On-Policy Preference Learning Driven By Synthetic Test Casescited by 8
37 more
  1. 2024LProtector: An LLM-driven Vulnerability Detection Systemcited by 8
  2. 2026ICAE-Bench: Evaluating Coding Agents as Interactive Project Builderscited by 6
  3. 2025CodeAssistBench (CAB): Dataset & Benchmarking for Multi-turn Chat-Based Code Assistancecited by 5
  4. 2025Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questionscited by 4
  5. 2025LPFQA: A Long-Tail Professional Forum-based Benchmark for LLM Evaluationcited by 4
  6. 2024Examination of Code generated by Large Language Modelscited by 3
  7. 2025Leveraging Metamemory Agent for Enhanced Data-Free Code Generation in Large Language Modelscited by 3
  8. 2026Humans are Missing from AI Coding Agent Researchcited by 3
  9. 2025Capability-Driven Skill Generation with LLMs: A RAG-Based Approach for Reusing Existing Libraries and Interfacescited by 2
  10. 2025JARVIS: A Multi-Agent Code Assistant for High-Quality EDA Script Generationcited by 2
  11. 2025TENET: One Step Toward Test-Driven Development for Repository-Level Code Generationcited by 2
  12. 2025SELF: A Robust Singular Value and Eigenvalue Approach for LLM Fingerprintingcited by 2
  13. 2025Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implicationscited by 2
  14. 2026ClarifySTL: An Interactive LLM Agent Framework for STL Transformation through Requirements Clarificationcited by 2
  15. 2025A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standardscited by 1
  16. 2025Breaking Memorization Barriers in LLM Code Fine-Tuning via Information Bottleneck for Improved Generalizationcited by 1
  17. 2026CAM: A Causality-based Analysis Framework for Multi-Agent Code Generation Systemscited by 1
  18. 2026FASE: Fast Adaptive Semantic Entropy for Code Qualitycited by 1
  19. 2026Automated Summarization of Software Documents: An LLM-based Multi-Agent Approachcited by 1
  20. 2025Mockingbird: How does LLM perform in general machine learning tasks?cited by 0
  21. 2025app.build: A Production Framework for Scaling Agentic Prompt-to-App Generation with Environment Scaffoldingcited by 0
  22. 2025When the Code Autopilot Breaks: Why LLMs Falter in Embedded Machine Learningcited by 0
  23. 2025Knowledge-Guided Multi-Agent Framework for Application-Level Software Code Generationcited by 0
  24. 2025Holistic Evaluation of State-of-the-Art LLMs for Code Generationcited by 0
  25. 2026LLM-Enabled Multi-Agent Systems: Empirical Evaluation and Insights into Emerging Design Patterns & Paradigmscited by 0
  26. 2026Design-Specification Tiling for ICL-based CAD Code Generationcited by 0
  27. 2026EffiPair: Improving the Efficiency of LLM-generated Code with Differential Execution Feedbackcited by 0
  28. 2026AI4BayesCode: From Natural Language Descriptions to Validated Modular Stateful Bayesian Samplerscited by 0
  29. 2026Quantize with Confidence? An Empirical Study of Quantization for Code Generationcited by 0
  30. 2026The Librarian Who Refused to Code: Model-Dependent Identity Enactment in LLM Code Generationcited by 0
  31. 2026CogEvol: Towards Efficient and Reliable Learning Environment Generationcited by 0
  32. 2026Propose to Learn, Learn to Propose: Evaluability-Aware Assistance under Bounded Rationalitycited by 0
  33. 2026CIT-CAD: Constraint Intent Tree-based CAD Code Generation and Verificationcited by 0
  34. 2026Test-Driven Approaches to Software Engineering with Large Language Models: A Survey of Phases, Tasks, and Agent Skillscited by 0
  35. 2026CONTRA: Discovering and Qualifying Behavior-Changing Questions for Selective Clarification in LLM Code Generationcited by 0
  36. 2026GTDD: Generative Test-Driven Development for AI Coding Agents with Adversarial Testingcited by 0
  37. 2026Passing Tests, Missing Semantics: Evidence of Interpretive Lock-In in Plan-Based DataFrame Code Generationcited by 0