# Unify and Triumph: Polyglot, Diverse, and Self-Consistent Generation of Unit Tests with LLMs

arXiv: https://arxiv.org/abs/2503.16144 · HTML: https://mlsanity.com/paper/2503.16144/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 20 Mar 2025.

## Builds on (most cited first, 23 of 23)

- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [CodeT: Code Generation with Generated Tests](https://mlsanity.com/paper/2207.10397/) (2022)
- [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023)
- [Software Testing with Large Language Models: Survey, Landscape, and Vision](https://mlsanity.com/paper/2307.07221/) (2023)
- [Productivity Assessment of Neural Code Completion](https://mlsanity.com/paper/2205.06537/) (2022)
- [Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT](https://mlsanity.com/paper/2304.10778/) (2023)
- [Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing](https://mlsanity.com/paper/2308.16557/) (2023)
- [Interactive Code Generation via Test-Driven User-Intent Formalization](https://mlsanity.com/paper/2208.05950/) (2022)
- [Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study](https://mlsanity.com/paper/2309.08221/) (2023)
- [ChatGPT for Vulnerability Detection, Classification, and Repair: How Far Are We?](https://mlsanity.com/paper/2310.09810/) (2023)
- [Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Code](https://mlsanity.com/paper/2206.01335/) (2022)
- [Large Language Models as Test Case Generators: Performance Evaluation and Enhancement](https://mlsanity.com/paper/2404.13340/) (2024)
- [Multi-modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2312.08762/) (2023)
- [Piloting Copilot, Codex, and StarCoder2: Hot Temperature, Cold Prompts, or Black Magic?](https://mlsanity.com/paper/2210.14699/) (2022)
- [Towards using Few-Shot Prompt Learning for Automating Model Completion](https://mlsanity.com/paper/2212.03404/) (2022)
- [An Empirical Study of Automated Unit Test Generation for Python](https://mlsanity.com/paper/2111.05003/) (2021)
- [TestSpark: IntelliJ IDEA's Ultimate Test Generation Companion](https://mlsanity.com/paper/2401.06580/) (2024)
- [On Codex Prompt Engineering for OCL Generation: An Empirical Study](https://mlsanity.com/paper/2303.16244/) (2023)
- [An empirical study of ChatGPT-3.5 on question answering and code maintenance](https://mlsanity.com/paper/2310.02104/) (2023)
- [Generative AI to Generate Test Data Generators](https://mlsanity.com/paper/2401.17626/) (2024)
- [Comparing Mutation Coverage Against Branch Coverage in an Industrial Setting](https://mlsanity.com/paper/2104.11767/) (2021)

## Abstract (arXiv)

Large language model (LLM)-based test generation has gained attention in software engineering, yet most studies evaluate LLMs' ability to generate unit tests in a single attempt for a given language, missing the opportunity to leverage LLM diversity for more robust testing. This paper introduces PolyTest, a novel approach that enhances test generation by exploiting polyglot and temperature-controlled diversity. PolyTest systematically leverages these properties in two complementary ways: (1) Cross-lingual test generation, where tests are generated in multiple languages at zero temperature and then unified; (2) Diverse test sampling, where multiple test sets are generated within the same language at a higher temperature before unification. A key insight is that LLMs can generate diverse yet contradicting tests -- same input, different expected outputs -- across languages and generations. PolyTest mitigates inconsistencies by unifying test sets, fostering self-consistency and improving overall test quality. Unlike single-language or single-attempt approaches, PolyTest enhances testing without requiring on-the-fly execution, making it particularly beneficial for weaker-performing languages. We evaluate PolyTest on Llama3-70B, GPT-4o, and GPT-3.5 using EvalPlus, generating tests in five languages (Java, C, Python, JavaScript, and a CSV-based format) at temperature 0 and sampling multiple sets at temperature 1. We observe that LLMs frequently generate contradicting tests across settings, and that PolyTest significantly improves test quality across all considered metrics -- number of tests, passing rate, statement/branch coverage (up to +9.01%), and mutation score (up to +11.23%). Finally, PolyTest outperforms Pynguin in test generation, passing rate, and mutation score.