mlsanityData to 7 Oct 2026

CoverUp: Effective High Coverage Test Generation for Python

arXiv 2403.16218 ↗First posted 24 Mar 2024cs.SE cs.AI cs.LGcited by 36

Not on the map. Why?

Cited by 20 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Testing is an essential part of software development. Test generation tools attempt to automate the otherwise labor-intensive task of test creation, but generating high-coverage tests remains challenging. This paper proposes CoverUp, a novel approach to driving the generation of high-coverage Python regression tests. CoverUp combines coverage analysis, code context, and feedback in prompts that iteratively guide the LLM to generate tests that improve line and branch coverage. We evaluate our prototype CoverUp implementation across a benchmark of challenging code derived from open-source Python projects and show that CoverUp substantially improves on the state of the art. Compared to CodaMosa, a hybrid search/LLM-based test generator, CoverUp achieves a per-module median line+branch coverage of 80% (vs. 47%). Compared to MuTAP, a mutation- and LLM-based test generator, CoverUp achieves an overall line+branch coverage of 89% (vs. 77%). We also demonstrate that CoverUp's performance stems not only from the LLM used but from the combined effectiveness of its components.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 18 of its 44 references that are in the map. First posted on arXiv 24 Mar 2024.

Builds on

18 of 44 references.

References · 18

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  3. 2023Large Language Modelscited by 774
  4. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  5. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 193
  6. 2023Software Testing with Large Language Models: Survey, Landscape, and Visioncited by 163
  7. 2020Unit Test Case Generation with Transformers and Focal Contextcited by 118
  8. 2023ChatUniTest: A Framework for LLM-Based Test Generationcited by 94
  9. 2024Automated Unit Test Improvement using Large Language Models at Metacited by 73
  10. 2024Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLMcited by 63
  11. 2023Effective Test Generation Using Pre-trained Large Language Models and Mutation Testingcited by 51
  12. 2022Interactive Code Generation via Test-Driven User-Intent Formalizationcited by 50
6 more
  1. 2022Pynguin: Automated Unit Test Generation for Pythoncited by 49
  2. 2022Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Codecited by 39
  3. 2024BugsInPy: A Database of Existing Bugs in Python Programs to Enable Controlled Testing and Debugging Studiescited by 35
  4. 2021An Empirical Study of Automated Unit Test Generation for Pythoncited by 10
  5. 2023SlipCover: Near Zero-Overhead Code Coverage for Pythoncited by 3
  6. 2023Do Automatic Test Generation Tools Generate Flaky Tests?cited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 10; 2026: 22.

36 papers.

Sort:

By area:

  1. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 400
  2. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 85
  3. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  4. 2025Learning to Generate Unit Test via Adversarial Reinforcement Learningcited by 11
  5. 2025TestForge: Feedback-Driven, Agentic Test Suite Generationcited by 8
  6. 2025Hallucination to Consensus: Multi-Agent LLMs for End-to-End JUnit Test Generationcited by 8
  7. 2026Test vs Mutant: Adversarial LLM Agents for Robust Unit Test Generationcited by 6
  8. 2026Code-A1: Adversarial Evolving of Code LLM and Test LLM via Reinforcement Learningcited by 6
  9. 2025Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesiscited by 4
  10. 2024Generative transformations and patterns in LLM-native approaches for software verification and falsificationcited by 2
  11. 2024Design choices made by LLM-based test generators prevent them from finding bugscited by 2
  12. 2025Large Language Models for Software Testing: A Research Roadmapcited by 2
24 more
  1. 2026SpecMind: Cognitively Inspired, Interactive Multi-Turn Framework for Postcondition Inferencecited by 2
  2. 2026VIBEPASS: Can Vibe Coders Really Pass the Vibe Check?cited by 2
  3. 2025R1-Fuzz: Specializing Language Models for Textual Fuzzing via Reinforcement Learningcited by 1
  4. 2025Navigating the Labyrinth: Path-Sensitive Unit Test Generation with Large Language Modelscited by 1
  5. 2025BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Softwarecited by 1
  6. 2025UnitTenX: Generating Tests for Legacy Packages with AI Agents Powered by Formal Verificationcited by 1
  7. 2026Planning to Explore: Curiosity-Driven Planning for LLM Test Generationcited by 1
  8. 2026MR-Coupler: Automated Metamorphic Test Generation via Functional Coupling Analysiscited by 1
  9. 2026Breaking, Stale, or Missing? Benchmarking Coding Agents on Project-Level Test Evolutioncited by 1
  10. 2026Teaching Code LLMs to Reason with Intermediate Formal Specificationscited by 1
  11. 2026SCATE: Learning to Supervise Coding Agents for Cost-Effective Test Generationcited by 1
  12. 2026ReProAgent: Tool-Augmented Multi-Stage Agentic Generation of Bug Reproduction Tests from Issue Reportscited by 1
  13. 2026Multi-Agent Coordinated Rename Refactoringcited by 0
  14. 2026CVeDRL: An Efficient Code Verifier via Difficulty-aware Reinforcement Learningcited by 0
  15. 2026AgileLog: A Forkable Shared Log for Agents on Data Streamscited by 0
  16. 2026ConCovUp: Effective Agent-Based Test Driver Generation for Concurrency Testingcited by 0
  17. 2026PBT-Bench: Benchmarking AI Agents on Property-Based Testingcited by 0
  18. 2026Prompt Coverage Adequacycited by 0
  19. 2026Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agentscited by 0
  20. 2026Rethinking Code Performance Benchmarks for LLMscited by 0
  21. 2026Test Coverage Analysis of Agentic Pull Requestscited by 0
  22. 2026Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Testscited by 0
  23. 2026Grounding AI Agents in Contracts: An Empirical Evaluation of Spec-Driven Test Generationcited by 0
  24. 2026Beyond Rule-Based Mutation Testing: Test-Aware Mutant Generation Using Large Language Modelscited by 0