# CAT-LM: Training Language Models on Aligned Code And Tests

arXiv: https://arxiv.org/abs/2310.01602 · HTML: https://mlsanity.com/paper/2310.01602/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 52 references that are in the map.
- First posted on arXiv 2 Oct 2023.

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [Efficient Training of Language Models to Fill in the Middle](https://mlsanity.com/paper/2207.14255/) (2022)
- [The Adverse Effects of Code Duplication in Machine Learning Models of Code](https://mlsanity.com/paper/1812.06469/) (2018)
- [Unit Test Case Generation with Transformers and Focal Context](https://mlsanity.com/paper/2009.05617/) (2020)
- [A Neural Model for Generating Natural Language Summaries of Program Subroutines](https://mlsanity.com/paper/1902.01954/) (2019)
- [TOGA: A Neural Method for Test Oracle Generation](https://mlsanity.com/paper/2109.09262/) (2021)
- [On Learning Meaningful Assert Statements for Unit Test Cases](https://mlsanity.com/paper/2002.05800/) (2020)
- [A Survey of Symbolic Execution Techniques](https://mlsanity.com/paper/1610.00502/) (2016)
- [Learning Deep Semantics for Test Completion](https://mlsanity.com/paper/2302.10166/) (2023)
- [ReAssert: Deep Learning for Assert Generation](https://mlsanity.com/paper/2011.09784/) (2020)
- [Developer-Centric Test Amplification The Interplay Between Automatic Generation and Human Exploration](https://mlsanity.com/paper/2108.12249/) (2021)

## Built on by (most cited first, 28 of 28)

- [Leveraging Large Language Models for Enhancing the Understandability of Generated Unit Tests](https://mlsanity.com/paper/2408.11710/) (2024)
- [Compute Optimal Scaling of Skills: Knowledge vs Reasoning](https://mlsanity.com/paper/2503.10061/) (2025)
- [RepoMasterEval: Evaluating Code Completion via Real-World Repositories](https://mlsanity.com/paper/2408.03519/) (2024)
- [B4: Towards Optimal Assessment of Plausible Code Solutions with Plausible Tests](https://mlsanity.com/paper/2409.08692/) (2024)
- [LLM Test Generation via Iterative Hybrid Program Analysis](https://mlsanity.com/paper/2503.13580/) (2025)
- [Hallucination to Consensus: Multi-Agent LLMs for End-to-End JUnit Test Generation](https://mlsanity.com/paper/2506.02943/) (2025)
- [Test vs Mutant: Adversarial LLM Agents for Robust Unit Test Generation](https://mlsanity.com/paper/2602.08146/) (2026)
- [UniTSyn: A Large-Scale Dataset Capable of Enhancing the Prowess of Large Language Models for Program Testing](https://mlsanity.com/paper/2402.03396/) (2024)
- [Evaluating the Ability of Large Language Models to Generate Verifiable Specifications in VeriFast](https://mlsanity.com/paper/2411.02318/) (2024)
- [exLong: Generating Exceptional Behavior Tests with Large Language Models](https://mlsanity.com/paper/2405.14619/) (2024)
- [Toward Automated Validation of Language Model Synthesized Test Cases using Semantic Entropy](https://mlsanity.com/paper/2411.08254/) (2024)
- [LlamaRestTest: Effective REST API Testing with Small Language Models](https://mlsanity.com/paper/2501.08598/) (2025)
- [Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesis](https://mlsanity.com/paper/2510.26423/) (2025)
- [FuzzAug: Data Augmentation by Coverage-guided Fuzzing for Neural Test Generation](https://mlsanity.com/paper/2406.08665/) (2024)
- [Generative transformations and patterns in LLM-native approaches for software verification and falsification](https://mlsanity.com/paper/2404.09384/) (2024)
- [Large Language Models for Software Testing: A Research Roadmap](https://mlsanity.com/paper/2509.25043/) (2025)
- [OptiHive: Ensemble Selection for LLM-Based Optimization via Statistical Modeling](https://mlsanity.com/paper/2508.02503/) (2025)
- [Navigating the Labyrinth: Path-Sensitive Unit Test Generation with Large Language Models](https://mlsanity.com/paper/2509.23812/) (2025)
- [SPELL: Synthesis of Programmatic Edits using LLMs](https://mlsanity.com/paper/2602.01107/) (2026)
- [Fuzzing with Agents? Generators Are All You Need](https://mlsanity.com/paper/2604.01442/) (2026)
- [Improving MPI Error Detection and Repair with Large Language Models and Bug References](https://mlsanity.com/paper/2604.02398/) (2026)
- [Program Structure-aware Language Models: Targeted Software Testing beyond Textual Semantics](https://mlsanity.com/paper/2604.17715/) (2026)
- [Inferring Code Correctness from Specification](https://mlsanity.com/paper/2605.29822/) (2026)
- [Rethinking Code Performance Benchmarks for LLMs](https://mlsanity.com/paper/2607.07619/) (2026)
- [Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)](https://mlsanity.com/paper/2607.22880/) (2026)
- [Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests](https://mlsanity.com/paper/2607.22883/) (2026)
- [Grounding AI Agents in Contracts: An Empirical Evaluation of Spec-Driven Test Generation](https://mlsanity.com/paper/2608.17177/) (2026)
- [Retrofitting Code Using LLMs to Support Exceptional Behavior](https://mlsanity.com/paper/2609.10397/) (2026)

## Abstract (arXiv)

Testing is an integral part of the software development process. Yet, writing tests is time-consuming and therefore often neglected. Classical test generation tools such as EvoSuite generate behavioral test suites by optimizing for coverage, but tend to produce tests that are hard to understand. Language models trained on code can generate code that is highly similar to that written by humans, but current models are trained to generate each file separately, as is standard practice in natural language processing, and thus fail to consider the code-under-test context when producing a test file. In this work, we propose the Aligned Code And Tests Language Model (CAT-LM), a GPT-style language model with 2.7 Billion parameters, trained on a corpus of Python and Java projects. We utilize a novel pretraining signal that explicitly considers the mapping between code and test files when available. We also drastically increase the maximum sequence length of inputs to 8,192 tokens, 4x more than typical code generation models, to ensure that the code context is available to the model when generating test code. We analyze its usefulness for realistic applications, showing that sampling with filtering (e.g., by compilability, coverage) allows it to efficiently produce tests that achieve coverage similar to ones written by developers while resembling their writing style. By utilizing the code context, CAT-LM generates more valid tests than even much larger language models trained with more data (CodeGen 16B and StarCoder) and substantially outperforms a recent test-specific model (TeCo) at test completion. Overall, our work highlights the importance of incorporating software-specific insights when training language models for code and paves the way to more powerful automated test generation.