mlsanityData to 14 Oct 2026

The Quality of Claude AI-authored Python Tests Is Not Weaker Than Human-authored Tests

arXiv 2608.15188 ↗First posted 15 Aug 2026cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We evaluate the quality of Claude AI-written Python tests against human-written Python tests from two established open-source projects Django and Pandas. Hundreds of tests per corpus are scored under one identical protocol. Using one-sided non-inferiority bounds, we find that the tests written by recent Claude models (Sonnet/Opus 4.6 and later) are no weaker than the two human-written corpora. In this study: (i) the AI-written corpus is tests from real tools, not synthetic tests generated in isolation against a fixed target, the setup used by every other AI-test-generation study we are aware of; (ii) every test is individually scored under three independent fault-injection protocols plus a seven-axis qualitative design rubric, allowing methods to cross-validate each other; (iii) tests are scored individually, rather than suite-level, identifying exactly which specific tests need attention.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 10 of its 30 references that are in the map. First posted on arXiv 15 Aug 2026.

Builds on

10 of 30 references.

References · 10

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  3. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 194
  4. 2024Do LLMs generate test oracles that capture the actual or the expected program behaviour?cited by 11
  5. 2016Assessing and Improving the Mutation Testing Practice of PITcited by 3
  6. 2021Does mutation testing improve testing practices?cited by 2
  7. 2026Evaluating LLM-Based Test Generation Under Software Evolutioncited by 2
  8. 2018Goal-Oriented Mutation Testing with Focal Methodscited by 1
  9. 2021Developers perception on the severity of test smells: an empirical studycited by 1
  10. 2026LLM vs. Human Unit Tests: Fault Detection on Real Python Bugscited by 1

Built on by

Not cited yet.