mlsanityData to 7 Oct 2026

Test-Driven Development for Code Generation

arXiv 2402.13521 ↗First posted 21 Feb 2024cs.SE cs.AIcited by 32

Not on the map. Why?

Abstract (arXiv)

Recent Large Language Models (LLMs) have demonstrated significant capabilities in generating code snippets directly from problem statements. This increasingly automated process mirrors traditional human-led software development, where code is often written in response to a requirement. Historically, Test-Driven Development (TDD) has proven its merit, requiring developers to write tests before the functional code, ensuring alignment with the initial problem statements. Applying TDD principles to LLM-based code generation offers one distinct benefit: it enables developers to verify the correctness of generated code against predefined tests. This paper investigates if and how TDD can be incorporated into AI-assisted code-generation processes. We experimentally evaluate our hypothesis that providing LLMs like GPT-4 and Llama 3 with tests in addition to the problem statements enhances code generation outcomes. We experimented with established function-level code generation benchmarks such as MBPP and HumanEval. Our results consistently demonstrate that including test cases leads to higher success in solving programming challenges. We assert that TDD is a promising paradigm for helping ensure that the code generated by LLMs effectively captures the requirements.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 18 of its 42 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

18 of 42 references.

References · 18

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  9. 2021Measuring Coding Challenge Competence With APPScited by 879
  10. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  11. 2022CodeT: Code Generation with Generated Testscited by 369
  12. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
6 more
  1. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  2. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  3. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 193
  4. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  5. 2022Interactive Code Generation via Test-Driven User-Intent Formalizationcited by 50
  6. 2022Compilable Neural Code Generation with Compiler Feedbackcited by 46

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 17; 2026: 14.

32 papers.

Sort:

By area:

  1. 2026SkillOps: Managing LLM Agent Skill Libraries as Self-Maintaining Software Ecosystemscited by 17
  2. 2025Large Language Models (LLMs) for Source Code Analysis: applications, models and datasetscited by 15
  3. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic Systemcited by 15
  4. 2025Beyond the Prompt: An Empirical Study of Cursor Rulescited by 7
  5. 2026TraceCoder: A Trace-Driven Multi-Agent Framework for Automated Debugging of LLM-Generated Codecited by 6
  6. 2025Automatically Generating Web Applications from Requirements Via Multi-Agent Test-Driven Developmentcited by 5
  7. 2026Are Coding Agents Generating Over-Mocked Tests? An Empirical Studycited by 5
  8. 2025CoreCodeBench: Decoupling Code Intelligence via Fine-Grained Repository-Level Taskscited by 3
  9. 2025Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questionscited by 3
  10. 2025Understanding and Bridging the Planner-Coder Gap: A Systematic Study on the Robustness of Multi-Agent Systems for Code Generationcited by 3
  11. 2026Automated structural testing of LLM-based agents: methods, framework, and case studiescited by 3
  12. 2024Design choices made by LLM-based test generators prevent them from finding bugscited by 2
20 more
  1. 2025TENET: One Step Toward Test-Driven Development for Repository-Level Code Generationcited by 2
  2. 2025Walking the Tightrope of LLMs for Software Development: A Practitioners' Perspectivecited by 2
  3. 2026Code for Machines, Not Just Humans: Quantifying AI-Friendliness with Code Health Metricscited by 2
  4. 2025Chain of Grounded Objectives: Bridging Process and Goal-oriented Prompting for Code Generationcited by 1
  5. 2025Understanding and supporting how developers prompt for LLM-powered code editing in practicecited by 1
  6. 2025Breaking Memorization Barriers in LLM Code Fine-Tuning via Information Bottleneck for Improved Generalizationcited by 1
  7. 2026OOM-RL: Out-of-Money Reinforcement Learning Market-Driven Alignment for LLM-Based Multi-Agent Systemscited by 1
  8. 2026TDD Governance for Multi-Agent Code Generation via Prompt Engineeringcited by 1
  9. 2026SemaDiff: Identifying Semantic-Changing Commits with Generated Code and Testscited by 1
  10. 2025Experimental Analysis of Productive Interaction Strategy with ChatGPT: User Study on Function and Project-level Code Generation Taskscited by 0
  11. 2025Leveraging Test Driven Development with Large Language Models for Reliable and Verifiable Spreadsheet Code Generation: A Research Frameworkcited by 0
  12. 2025A Comparison of Conversational Models and Humans in Answering Technical Questions: the Firefox Casecited by 0
  13. 2025NALA_MAINZ at BLP-2025 Task 2: A Multi-agent Approach for Bangla Instruction to Python Code Generationcited by 0
  14. 2025Chain of Unit-Physics: A Primitive-Centric Approach to Scientific Code Synthesiscited by 0
  15. 2026Advancing Language Models for Code-related Taskscited by 0
  16. 2026Inferring Code Correctness from Specificationcited by 0
  17. 2026An Exploratory Case Study of LLM-Assisted Refactoring and Gameplay Feature Generation in an Endless Runner Gamecited by 0
  18. 2026Prompt Coach: An Empirical Evaluation of an Agentic Tutor for Learning Prompt Engineering in Software Developmentcited by 0
  19. 2026MineValiCoder: Reliable Code Generation with Test Case Quality Mining and Bipartite Graph-Based Mutual Validationcited by 0
  20. 2026Retrofitting Code Using LLMs to Support Exceptional Behaviorcited by 0