# CoverUp: Effective High Coverage Test Generation for Python

arXiv: https://arxiv.org/abs/2403.16218 · HTML: https://mlsanity.com/paper/2403.16218/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 44 references that are in the map.
- First posted on arXiv 24 Mar 2024.

## Builds on (most cited first, 18 of 18)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023)
- [Software Testing with Large Language Models: Survey, Landscape, and Vision](https://mlsanity.com/paper/2307.07221/) (2023)
- [Unit Test Case Generation with Transformers and Focal Context](https://mlsanity.com/paper/2009.05617/) (2020)
- [ChatUniTest: A Framework for LLM-Based Test Generation](https://mlsanity.com/paper/2305.04764/) (2023)
- [Automated Unit Test Improvement using Large Language Models at Meta](https://mlsanity.com/paper/2402.09171/) (2024)
- [Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLM](https://mlsanity.com/paper/2402.00097/) (2024)
- [Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing](https://mlsanity.com/paper/2308.16557/) (2023)
- [Pynguin: Automated Unit Test Generation for Python](https://mlsanity.com/paper/2202.05218/) (2022)
- [Interactive Code Generation via Test-Driven User-Intent Formalization](https://mlsanity.com/paper/2208.05950/) (2022)
- [Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Code](https://mlsanity.com/paper/2206.01335/) (2022)
- [BugsInPy: A Database of Existing Bugs in Python Programs to Enable Controlled Testing and Debugging Studies](https://mlsanity.com/paper/2401.15481/) (2024)
- [An Empirical Study of Automated Unit Test Generation for Python](https://mlsanity.com/paper/2111.05003/) (2021)
- [SlipCover: Near Zero-Overhead Code Coverage for Python](https://mlsanity.com/paper/2305.02886/) (2023)
- [Do Automatic Test Generation Tools Generate Flaky Tests?](https://mlsanity.com/paper/2310.05223/) (2023)

## Built on by (most cited first, 30 of 38)

- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future](https://mlsanity.com/paper/2408.02479/) (2024)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [Learning to Generate Unit Test via Adversarial Reinforcement Learning](https://mlsanity.com/paper/2508.21107/) (2025)
- [TestForge: Feedback-Driven, Agentic Test Suite Generation](https://mlsanity.com/paper/2503.14713/) (2025)
- [Hallucination to Consensus: Multi-Agent LLMs for End-to-End JUnit Test Generation](https://mlsanity.com/paper/2506.02943/) (2025)
- [Code-A1: Adversarial Evolving of Code LLM and Test LLM via Reinforcement Learning](https://mlsanity.com/paper/2603.15611/) (2026)
- [Test vs Mutant: Adversarial LLM Agents for Robust Unit Test Generation](https://mlsanity.com/paper/2602.08146/) (2026)
- [Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesis](https://mlsanity.com/paper/2510.26423/) (2025)
- [Generative transformations and patterns in LLM-native approaches for software verification and falsification](https://mlsanity.com/paper/2404.09384/) (2024)
- [Design choices made by LLM-based test generators prevent them from finding bugs](https://mlsanity.com/paper/2412.14137/) (2024)
- [Large Language Models for Software Testing: A Research Roadmap](https://mlsanity.com/paper/2509.25043/) (2025)
- [BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Software](https://mlsanity.com/paper/2509.25248/) (2025)
- [SpecMind: Cognitively Inspired, Interactive Multi-Turn Framework for Postcondition Inference](https://mlsanity.com/paper/2602.20610/) (2026)
- [VIBEPASS: Can Vibe Coders Really Pass the Vibe Check?](https://mlsanity.com/paper/2603.15921/) (2026)
- [R1-Fuzz: Specializing Language Models for Textual Fuzzing via Reinforcement Learning](https://mlsanity.com/paper/2509.20384/) (2025)
- [Navigating the Labyrinth: Path-Sensitive Unit Test Generation with Large Language Models](https://mlsanity.com/paper/2509.23812/) (2025)
- [UnitTenX: Generating Tests for Legacy Packages with AI Agents Powered by Formal Verification](https://mlsanity.com/paper/2510.05441/) (2025)
- [CVeDRL: An Efficient Code Verifier via Difficulty-aware Reinforcement Learning](https://mlsanity.com/paper/2601.22803/) (2026)
- [Planning to Explore: Curiosity-Driven Planning for LLM Test Generation](https://mlsanity.com/paper/2604.05159/) (2026)
- [MR-Coupler: Automated Metamorphic Test Generation via Functional Coupling Analysis](https://mlsanity.com/paper/2604.10126/) (2026)
- [Breaking, Stale, or Missing? Benchmarking Coding Agents on Project-Level Test Evolution](https://mlsanity.com/paper/2605.06125/) (2026)
- [PBT-Bench: Benchmarking AI Agents on Property-Based Testing](https://mlsanity.com/paper/2605.15229/) (2026)
- [Teaching Code LLMs to Reason with Intermediate Formal Specifications](https://mlsanity.com/paper/2607.04232/) (2026)
- [SCATE: Learning to Supervise Coding Agents for Cost-Effective Test Generation](https://mlsanity.com/paper/2607.08983/) (2026)
- [ReProAgent: Tool-Augmented Multi-Stage Agentic Generation of Bug Reproduction Tests from Issue Reports](https://mlsanity.com/paper/2607.09123/) (2026)
- [Multi-Agent Coordinated Rename Refactoring](https://mlsanity.com/paper/2601.00482/) (2026)
- [AgileLog: A Forkable Shared Log for Agents on Data Streams](https://mlsanity.com/paper/2604.14590/) (2026)
- [ConCovUp: Effective Agent-Based Test Driver Generation for Concurrency Testing](https://mlsanity.com/paper/2605.09573/) (2026)
- [Prompt Coverage Adequacy](https://mlsanity.com/paper/2607.02057/) (2026)

## Abstract (arXiv)

Testing is an essential part of software development. Test generation tools attempt to automate the otherwise labor-intensive task of test creation, but generating high-coverage tests remains challenging. This paper proposes CoverUp, a novel approach to driving the generation of high-coverage Python regression tests. CoverUp combines coverage analysis, code context, and feedback in prompts that iteratively guide the LLM to generate tests that improve line and branch coverage. We evaluate our prototype CoverUp implementation across a benchmark of challenging code derived from open-source Python projects and show that CoverUp substantially improves on the state of the art. Compared to CodaMosa, a hybrid search/LLM-based test generator, CoverUp achieves a per-module median line+branch coverage of 80% (vs. 47%). Compared to MuTAP, a mutation- and LLM-based test generator, CoverUp achieves an overall line+branch coverage of 89% (vs. 77%). We also demonstrate that CoverUp's performance stems not only from the LLM used but from the combined effectiveness of its components.