mlsanityData to 7 Oct 2026

Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study

arXiv 2403.08604 ↗First posted 13 Mar 2024cs.CL cs.SEcited by 40

Not on the map. Why?

Cited by 24 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advancements in large language models (LLMs) have significantly enhanced their coding capabilities. However, existing benchmarks predominantly focused on simplified or isolated aspects of coding, such as single-file code generation or repository issue debugging, falling short of measuring the full spectrum of challenges raised by real-world programming activities. In this case study, we explore the performance of LLMs across the entire software development lifecycle with DevEval, encompassing stages including software design, environment setup, implementation, acceptance testing, and unit testing. DevEval features four programming languages, multiple domains, high-quality data collection, and carefully designed and verified metrics for each task. Empirical studies show that current LLMs, including GPT-4, fail to solve the challenges presented within DevEval. Our findings offer actionable insights for the future development of LLMs toward real-world programming applications.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 20 of its 34 references that are in the map. First posted on arXiv 13 Mar 2024.

Builds on

20 of 34 references.

References · 20

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  6. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  7. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  8. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  9. 2021Measuring Coding Challenge Competence With APPScited by 879
  10. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  11. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  12. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
8 more
  1. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  2. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  3. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  4. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  5. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  6. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 89
  7. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 86
  8. 2023Experiential Co-Learning of Software-Developing Agentscited by 57

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 11; 2026: 24.

40 papers.

Sort:

By area:

  1. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  2. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 161
  3. 2026FeatureBench: Benchmarking Agentic Coding for Complex Feature Developmentcited by 30
  4. 2025RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Codecited by 28
  5. 2025ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generationcited by 26
  6. 2026SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agentscited by 25
  7. 2024PyBench: Evaluating LLM Agent on various real-world coding taskscited by 24
  8. 2025CRUST-Bench: A Comprehensive Benchmark for C-to-safe-Rust Transpilationcited by 19
  9. 2025LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineeringcited by 16
  10. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 15
  11. 2026TraceLab: Characterizing Coding Agent Workloads for LLM Servingcited by 14
  12. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
28 more
  1. 2024Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Modelscited by 9
  2. 2026DevOps-Gym: Benchmarking AI Agents in Software DevOps Cyclecited by 9
  3. 2026GameGen-Verifier: Parallel Keypoint-Based Verification for LLM-Generated Games via Runtime State Injectioncited by 9
  4. 2025The Impact of Generative AI Coding Assistants on Developers Who Are Visually Impairedcited by 7
  5. 2025Towards Realistic Project-Level Code Generation via Multi-Agent Collaboration and Semantic Architecture Modelingcited by 6
  6. 2026ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Developmentcited by 6
  7. 2026When Parallelism Pays Off: Cohesion-Aware Task Partitioning for Multi-Agent Codingcited by 4
  8. 2025CoreCodeBench: Decoupling Code Intelligence via Fine-Grained Repository-Level Taskscited by 3
  9. 2025PerfBench: Can Agents Resolve Real-World Performance Bugs?cited by 3
  10. 2026SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineeringcited by 3
  11. 2026Humans are Missing from AI Coding Agent Researchcited by 3
  12. 2026RM -RF: Reward Model for Run-Free Unit Test Evaluationcited by 2
  13. 2026CUBE: A Standard for Unifying Agent Benchmarkscited by 2
  14. 2026Credit-Budgeted ICPC-Style Coding: When Agents Must Pay for Every Decisioncited by 2
  15. 2026Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QAcited by 2
  16. 2026Repo0: Design-Driven Zero-to-All Code Generationcited by 2
  17. 2024Effective Large Language Model Debugging with Best-first Tree Searchcited by 1
  18. 2026AlphaEval: Evaluating Agents in Productioncited by 1
  19. 2026Do programming languages still matter to your AI coding agent teammate? Evidence at scale from chess enginescited by 1
  20. 2026Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluationcited by 1
  21. 2025PIPer: On-Device Environment Setup via Online Reinforcement Learningcited by 0
  22. 2025Benchmarking LLM Agents for Wealth-Management Workflowscited by 0
  23. 2026TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenancecited by 0
  24. 2026From What to How: Bridging User Requirements with Software Development Using Large Language Modelscited by 0
  25. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutionscited by 0
  26. 2026AgentRoom: Concurrent Multi-Agent Coding in a CRDT-Backed Shared Workspacecited by 0
  27. 2026ττ-Bench: An Environment for End-To-End, Realistic Agent Constructioncited by 0
  28. 2026E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratchcited by 0