# Can Large Language Models Write Good Property-Based Tests?

arXiv: https://arxiv.org/abs/2307.04346 · HTML: https://mlsanity.com/paper/2307.04346/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jul 2023.

## Built on by (most cited first, 30 of 31)

- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation](https://mlsanity.com/paper/2308.01861/) (2023)
- [Fuzz4All: Universal Fuzzing with Large Language Models](https://mlsanity.com/paper/2308.04748/) (2023)
- [Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code](https://mlsanity.com/paper/2311.07989/) (2023)
- [Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?](https://mlsanity.com/paper/2310.01831/) (2023)
- [Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generation](https://mlsanity.com/paper/2308.01240/) (2023)
- [TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?](https://mlsanity.com/paper/2412.02883/) (2024)
- [Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation](https://mlsanity.com/paper/2310.02368/) (2023)
- [Text-to-CAD Evaluation with CADTests](https://mlsanity.com/paper/2605.07807/) (2026)
- [Turbulence: Systematically and Automatically Testing Instruction-Tuned Large Language Models for Code](https://mlsanity.com/paper/2312.14856/) (2023)
- [Agentic Property-Based Testing: Finding Bugs Across the Python Ecosystem](https://mlsanity.com/paper/2510.09907/) (2025)
- [Does Few-Shot Learning Help LLM Performance in Code Synthesis?](https://mlsanity.com/paper/2412.02906/) (2024)
- [Using Large Language Models for Student-Code Guided Test Case Generation in Computer Science Education](https://mlsanity.com/paper/2402.07081/) (2024)
- [Toward Automated Validation of Language Model Synthesized Test Cases using Semantic Entropy](https://mlsanity.com/paper/2411.08254/) (2024)
- [Code Copycat Conundrum: Demystifying Repetition in LLM-based Code Generation](https://mlsanity.com/paper/2504.12608/) (2025)
- [PBFuzz: Agentic Directed Fuzzing for PoV Generation](https://mlsanity.com/paper/2512.04611/) (2025)
- [Program Skeletons for Automated Program Translation](https://mlsanity.com/paper/2504.07483/) (2025)
- [VeriContest: A Competitive-Programming Benchmark for Verifiable Code Generation](https://mlsanity.com/paper/2605.08553/) (2026)
- [S3LLM: Large-Scale Scientific Software Understanding with LLMs using Source, Metadata, and Document](https://mlsanity.com/paper/2403.10588/) (2024)
- [LLMs in the Heart of Differential Testing: A Case Study on a Medical Rule Engine](https://mlsanity.com/paper/2404.03664/) (2024)
- [Generative transformations and patterns in LLM-native approaches for software verification and falsification](https://mlsanity.com/paper/2404.09384/) (2024)
- [Effective LLM Code Refinement via Property-Oriented and Structurally Minimal Feedback](https://mlsanity.com/paper/2506.18315/) (2025)
- [Large Language Models for Software Testing: A Research Roadmap](https://mlsanity.com/paper/2509.25043/) (2025)
- [SpecMind: Cognitively Inspired, Interactive Multi-Turn Framework for Postcondition Inference](https://mlsanity.com/paper/2602.20610/) (2026)
- [A Systematic Approach for Assessing Large Language Models' Test Case Generation Capability](https://mlsanity.com/paper/2502.02866/) (2025)
- [PBT-Bench: Benchmarking AI Agents on Property-Based Testing](https://mlsanity.com/paper/2605.15229/) (2026)
- [Teaching Code LLMs to Reason with Intermediate Formal Specifications](https://mlsanity.com/paper/2607.04232/) (2026)
- [CodeScholar: Growing Idiomatic Code Examples](https://mlsanity.com/paper/2312.15157/) (2023)
- [Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation](https://mlsanity.com/paper/2412.14308/) (2024)
- [Fuzzing with Agents? Generators Are All You Need](https://mlsanity.com/paper/2604.01442/) (2026)

## Abstract (arXiv)

Property-based testing (PBT), while an established technique in the software testing research community, is still relatively underused in real-world software. Pain points in writing property-based tests include implementing diverse random input generators and thinking of meaningful properties to test. Developers, however, are more amenable to writing documentation; plenty of library API documentation is available and can be used as natural language specifications for PBTs. As large language models (LLMs) have recently shown promise in a variety of coding tasks, we investigate using modern LLMs to automatically synthesize PBTs using two prompting techniques. A key challenge is to rigorously evaluate the LLM-synthesized PBTs. We propose a methodology to do so considering several properties of the generated tests: (1) validity, (2) soundness, and (3) property coverage, a novel metric that measures the ability of the PBT to detect property violations through generation of property mutants. In our evaluation on 40 Python library API methods across three models (GPT-4, Gemini-1.5-Pro, Claude-3-Opus), we find that with the best model and prompting approach, a valid and sound PBT can be synthesized in 2.4 samples on average. We additionally find that our metric for determining soundness of a PBT is aligned with human judgment of property assertions, achieving a precision of 100% and recall of 97%. Finally, we evaluate the property coverage of LLMs across all API methods and find that the best model (GPT-4) is able to automatically synthesize correct PBTs for 21% of properties extractable from API documentation.