# LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation

arXiv: https://arxiv.org/abs/2404.10100 · HTML: https://mlsanity.com/paper/2404.10100/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Apr 2024.

## Built on by (most cited first, 30 of 49)

- [From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future](https://mlsanity.com/paper/2408.02479/) (2024)
- [SpecRover: Code Intent Extraction via LLMs](https://mlsanity.com/paper/2408.02232/) (2024)
- [Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI](https://mlsanity.com/paper/2505.19443/) (2025)
- [Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering](https://mlsanity.com/paper/2502.13069/) (2025)
- [AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities](https://mlsanity.com/paper/2508.11126/) (2025)
- [HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agent](https://mlsanity.com/paper/2406.00215/) (2024)
- [debug-gym: A Text-Based Environment for Interactive Debugging](https://mlsanity.com/paper/2503.21557/) (2025)
- [Large Language Models (LLMs) for Source Code Analysis: applications, models and datasets](https://mlsanity.com/paper/2503.17502/) (2025)
- [APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Serving](https://mlsanity.com/paper/2411.17651/) (2024)
- [Tool-integrated Reinforcement Learning for Repo Deep Search](https://mlsanity.com/paper/2508.03012/) (2025)
- [Unified attacks to large language model watermarks: spoofing and scrubbing in unauthorized knowledge distillation](https://mlsanity.com/paper/2504.17480/) (2025)
- [PLUM: Improving Code LMs with Execution-Guided On-Policy Preference Learning Driven By Synthetic Test Cases](https://mlsanity.com/paper/2406.06887/) (2024)
- [LProtector: An LLM-driven Vulnerability Detection System](https://mlsanity.com/paper/2411.06493/) (2024)
- [ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders](https://mlsanity.com/paper/2607.21217/) (2026)
- [CodeAssistBench (CAB): Dataset & Benchmarking for Multi-turn Chat-Based Code Assistance](https://mlsanity.com/paper/2507.10646/) (2025)
- [Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions](https://mlsanity.com/paper/2507.21285/) (2025)
- [LPFQA: A Long-Tail Professional Forum-based Benchmark for LLM Evaluation](https://mlsanity.com/paper/2511.06346/) (2025)
- [Examination of Code generated by Large Language Models](https://mlsanity.com/paper/2408.16601/) (2024)
- [Leveraging Metamemory Agent for Enhanced Data-Free Code Generation in Large Language Models](https://mlsanity.com/paper/2501.07892/) (2025)
- [Humans are Missing from AI Coding Agent Research](https://mlsanity.com/paper/2608.12355/) (2026)
- [Capability-Driven Skill Generation with LLMs: A RAG-Based Approach for Reusing Existing Libraries and Interfaces](https://mlsanity.com/paper/2505.03295/) (2025)
- [JARVIS: A Multi-Agent Code Assistant for High-Quality EDA Script Generation](https://mlsanity.com/paper/2505.14978/) (2025)
- [TENET: One Step Toward Test-Driven Development for Repository-Level Code Generation](https://mlsanity.com/paper/2509.24148/) (2025)
- [SELF: A Robust Singular Value and Eigenvalue Approach for LLM Fingerprinting](https://mlsanity.com/paper/2512.03620/) (2025)
- [Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implications](https://mlsanity.com/paper/2601.08833/) (2025)
- [ClarifySTL: An Interactive LLM Agent Framework for STL Transformation through Requirements Clarification](https://mlsanity.com/paper/2605.01209/) (2026)
- [A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standards](https://mlsanity.com/paper/2505.13766/) (2025)
- [Breaking Memorization Barriers in LLM Code Fine-Tuning via Information Bottleneck for Improved Generalization](https://mlsanity.com/paper/2510.16022/) (2025)
- [CAM: A Causality-based Analysis Framework for Multi-Agent Code Generation Systems](https://mlsanity.com/paper/2602.02138/) (2026)
- [FASE: Fast Adaptive Semantic Entropy for Code Quality](https://mlsanity.com/paper/2606.09800/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown great potential in automating significant aspects of coding by producing natural code from informal natural language (NL) intent. However, given NL is informal, it does not lend easily to checking that the generated code correctly satisfies the user intent. In this paper, we propose a novel interactive workflow TiCoder for guided intent clarification (i.e., partial formalization) through tests to support the generation of more accurate code suggestions. Through a mixed methods user study with 15 programmers, we present an empirical evaluation of the effectiveness of the workflow to improve code generation accuracy. We find that participants using the proposed workflow are significantly more likely to correctly evaluate AI generated code, and report significantly less task-induced cognitive load. Furthermore, we test the potential of the workflow at scale with four different state-of-the-art LLMs on two python datasets, using an idealized proxy for a user feedback. We observe an average absolute improvement of 45.97% in the pass@1 code generation accuracy for both datasets and across all LLMs within 5 user interactions, in addition to the automatic generation of accompanying unit tests.