# Learning Deep Semantics for Test Completion

arXiv: https://arxiv.org/abs/2302.10166 · HTML: https://mlsanity.com/paper/2302.10166/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 101 references that are in the map.
- First posted on arXiv 20 Feb 2023.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020)
- [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023)
- [A Transformer-based Approach for Source Code Summarization](https://mlsanity.com/paper/2005.00653/) (2020)
- [Natural Language to Code Translation with Execution](https://mlsanity.com/paper/2204.11454/) (2022)
- [Unit Test Case Generation with Transformers and Focal Context](https://mlsanity.com/paper/2009.05617/) (2020)
- [NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System](https://mlsanity.com/paper/1802.08979/) (2018)
- [A Neural Model for Generating Natural Language Summaries of Program Subroutines](https://mlsanity.com/paper/1902.01954/) (2019)
- [Code Generation as a Dual Task of Code Summarization](https://mlsanity.com/paper/1910.05923/) (2019)
- [TOGA: A Neural Method for Test Oracle Generation](https://mlsanity.com/paper/2109.09262/) (2021)
- [Pythia: AI-assisted Code Completion System](https://mlsanity.com/paper/1912.00742/) (2019)
- [Code Completion with Neural Attention and Pointer Networks](https://mlsanity.com/paper/1711.09573/) (2017)
- [Reasoning Like Program Executors](https://mlsanity.com/paper/2201.11473/) (2022)
- [CoditT5: Pretraining for Source Code and Natural Language Editing](https://mlsanity.com/paper/2208.05446/) (2022)
- [On Learning Meaningful Assert Statements for Unit Test Cases](https://mlsanity.com/paper/2002.05800/) (2020)

## Built on by (most cited first, 30 of 31)

- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Can LLMs Replace Manual Annotation of Software Engineering Artifacts?](https://mlsanity.com/paper/2408.05534/) (2024)
- [WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models](https://mlsanity.com/paper/2310.15991/) (2023)
- [CAT-LM: Training Language Models on Aligned Code And Tests](https://mlsanity.com/paper/2310.01602/) (2023)
- [KernelGPT: Enhanced Kernel Fuzzing via Large Language Models](https://mlsanity.com/paper/2401.00563/) (2023)
- [B4: Towards Optimal Assessment of Plausible Code Solutions with Plausible Tests](https://mlsanity.com/paper/2409.08692/) (2024)
- [TestForge: Feedback-Driven, Agentic Test Suite Generation](https://mlsanity.com/paper/2503.14713/) (2025)
- [A System for Automated Unit Test Generation Using Large Language Models and Assessment of Generated Test Suites](https://mlsanity.com/paper/2408.07846/) (2024)
- [CLOVER: A Test Case Generation Benchmark with Coverage, Long-Context, and Verification](https://mlsanity.com/paper/2502.08806/) (2025)
- [Domain Adaptation for Code Model-based Unit Test Case Generation](https://mlsanity.com/paper/2308.08033/) (2023)
- [Assessing Evaluation Metrics for Neural Test Oracle Generation](https://mlsanity.com/paper/2310.07856/) (2023)
- [InfCode: Adversarial Iterative Refinement of Tests and Patches for Reliable Software Issue Resolution](https://mlsanity.com/paper/2511.16004/) (2025)
- [Test vs Mutant: Adversarial LLM Agents for Robust Unit Test Generation](https://mlsanity.com/paper/2602.08146/) (2026)
- [Combining Large Language Models with Static Analyzers for Code Review Generation](https://mlsanity.com/paper/2502.06633/) (2025)
- [UniTSyn: A Large-Scale Dataset Capable of Enhancing the Prowess of Large Language Models for Program Testing](https://mlsanity.com/paper/2402.03396/) (2024)
- [An Empirical Study of AI-based Smart Contract Creation](https://mlsanity.com/paper/2308.02955/) (2023)
- [LLM4TDD: Best Practices for Test Driven Development Using Large Language Models](https://mlsanity.com/paper/2312.04687/) (2023)
- [exLong: Generating Exceptional Behavior Tests with Large Language Models](https://mlsanity.com/paper/2405.14619/) (2024)
- [FuzzAug: Data Augmentation by Coverage-guided Fuzzing for Neural Test Generation](https://mlsanity.com/paper/2406.08665/) (2024)
- [Treefix: Enabling Execution with a Tree of Prefixes](https://mlsanity.com/paper/2501.12339/) (2025)
- [A Tool for Generating Exceptional Behavior Tests With Large Language Models](https://mlsanity.com/paper/2505.22818/) (2025)
- [Large Language Models for Software Testing: A Research Roadmap](https://mlsanity.com/paper/2509.25043/) (2025)
- [Navigating the Labyrinth: Path-Sensitive Unit Test Generation with Large Language Models](https://mlsanity.com/paper/2509.23812/) (2025)
- [LLMs for Automated Unit Test Generation and Assessment in Java: The AgoneTest Framework](https://mlsanity.com/paper/2511.20403/) (2025)
- [Compiling Large Multi-Modal Requirement Documents into Runnable Software Systems: From an Agentic Test-Driven Perspective](https://mlsanity.com/paper/2602.13723/) (2026)
- [AutoTestForge: A Multidimensional Automated Testing Framework for Natural Language Processing Models](https://mlsanity.com/paper/2503.05102/) (2025)
- [SAGE:Specification-Aware Grammar Extraction for Automated Test Case Generation with LLMs](https://mlsanity.com/paper/2506.11081/) (2025)
- [Improving MPI Error Detection and Repair with Large Language Models and Bug References](https://mlsanity.com/paper/2604.02398/) (2026)
- [Agent-Based Test Assertion Generation via Diverse Perspective Aggregation](https://mlsanity.com/paper/2608.05822/) (2026)
- [Grounding AI Agents in Contracts: An Empirical Evaluation of Spec-Driven Test Generation](https://mlsanity.com/paper/2608.17177/) (2026)

## Abstract (arXiv)

Writing tests is a time-consuming yet essential task during software development. We propose to leverage recent advances in deep learning for text and code generation to assist developers in writing tests. We formalize the novel task of test completion to automatically complete the next statement in a test method based on the context of prior statements and the code under test. We develop TeCo -- a deep learning model using code semantics for test completion. The key insight underlying TeCo is that predicting the next statement in a test method requires reasoning about code execution, which is hard to do with only syntax-level data that existing code completion models use. TeCo extracts and uses six kinds of code semantics data, including the execution result of prior statements and the execution context of the test method. To provide a testbed for this new task, as well as to evaluate TeCo, we collect a corpus of 130,934 test methods from 1,270 open-source Java projects. Our results show that TeCo achieves an exact-match accuracy of 18, which is 29% higher than the best baseline using syntax-level data only. When measuring functional correctness of generated next statement, TeCo can generate runnable code in 29% of the cases compared to 18% obtained by the best baseline. Moreover, TeCo is significantly better than prior work on test oracle generation.