# Unit Test Case Generation with Transformers and Focal Context

arXiv: https://arxiv.org/abs/2009.05617 · HTML: https://mlsanity.com/paper/2009.05617/

## Key facts

- Cited by 118 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Sep 2020.

## Built on by (most cited first, 30 of 118)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [CodeT: Code Generation with Generated Tests](https://mlsanity.com/paper/2207.10397/) (2022)
- [LEVER: Learning to Verify Language-to-Code Generation with Execution](https://mlsanity.com/paper/2302.08468/) (2023)
- [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023)
- [Planning with Large Language Models for Code Generation](https://mlsanity.com/paper/2303.05510/) (2023)
- [SantaCoder: don't reach for the stars!](https://mlsanity.com/paper/2301.03988/) (2023)
- [MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation](https://mlsanity.com/paper/2208.08227/) (2022)
- [SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents](https://mlsanity.com/paper/2406.12952/) (2024)
- [Learning Performance-Improving Code Edits](https://mlsanity.com/paper/2302.07867/) (2023)
- [Execution-Based Evaluation for Open-Domain Code Generation](https://mlsanity.com/paper/2212.10481/) (2022)
- [From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future](https://mlsanity.com/paper/2408.02479/) (2024)
- [Leveraging Automated Unit Tests for Unsupervised Code Translation](https://mlsanity.com/paper/2110.06773/) (2021)
- [Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLM](https://mlsanity.com/paper/2402.00097/) (2024)
- [Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?](https://mlsanity.com/paper/2310.01831/) (2023)
- [Exploring and Unleashing the Power of Large Language Models in Automated Code Translation](https://mlsanity.com/paper/2404.14646/) (2024)
- [Co-Evolving LLM Coder and Unit Tester via Reinforcement Learning](https://mlsanity.com/paper/2506.03136/) (2025)
- [Using Large Language Models to Generate JUnit Tests: An Empirical Study](https://mlsanity.com/paper/2305.00418/) (2023)
- [The Limits of Inference Scaling Through Resampling](https://mlsanity.com/paper/2411.17501/) (2024)
- [Interactive Code Generation via Test-Driven User-Intent Formalization](https://mlsanity.com/paper/2208.05950/) (2022)
- [Investigating Explainability of Generative AI for Code through Scenario-based Design](https://mlsanity.com/paper/2202.04903/) (2022)
- [CoverUp: Effective High Coverage Test Generation for Python](https://mlsanity.com/paper/2403.16218/) (2024)
- [TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?](https://mlsanity.com/paper/2412.02883/) (2024)
- [Learning Deep Semantics for Test Completion](https://mlsanity.com/paper/2302.10166/) (2023)
- [CAT-LM: Training Language Models on Aligned Code And Tests](https://mlsanity.com/paper/2310.01602/) (2023)
- [Large Language Models as Test Case Generators: Performance Evaluation and Enhancement](https://mlsanity.com/paper/2404.13340/) (2024)
- [HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agent](https://mlsanity.com/paper/2406.00215/) (2024)
- [CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarks](https://mlsanity.com/paper/2404.00566/) (2024)

## Abstract (arXiv)

Automated unit test case generation tools facilitate test-driven development and support developers by suggesting tests intended to identify flaws in their code. Existing approaches are usually guided by the test coverage criteria, generating synthetic test cases that are often difficult for developers to read or understand. In this paper we propose AthenaTest, an approach that aims to generate unit test cases by learning from real-world focal methods and developer-written testcases. We formulate unit test case generation as a sequence-to-sequence learning task, adopting a two-step training procedure consisting of denoising pretraining on a large unsupervised Java corpus, and supervised finetuning for a downstream translation task of generating unit tests. We investigate the impact of natural language and source code pretraining, as well as the focal context information surrounding the focal method. Both techniques provide improvements in terms of validation loss, with pretraining yielding 25% relative improvement and focal context providing additional 11.1% improvement. We also introduce Methods2Test, the largest publicly available supervised parallel corpus of unit test case methods and corresponding focal methods in Java, which comprises 780K test cases mined from 91K open-source repositories from GitHub. We evaluate AthenaTest on five defects4j projects, generating 25K passing test cases covering 43.7% of the focal methods with only 30 attempts. We execute the test cases, collect test coverage information, and compare them with test cases generated by EvoSuite and GPT-3, finding that our approach outperforms GPT-3 and has comparable coverage w.r.t. EvoSuite. Finally, we survey professional developers on their preference in terms of readability, understandability, and testing effectiveness of the generated tests, showing overwhelmingly preference towards AthenaTest.