# Can ChatGPT advance software testing intelligence? An experience report on metamorphic testing

arXiv: https://arxiv.org/abs/2310.19204 · HTML: https://mlsanity.com/paper/2310.19204/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 9 references that are in the map.
- First posted on arXiv 30 Oct 2023.

## Builds on (most cited first, 3 of 3)

- [A Categorical Archive of ChatGPT Failures](https://mlsanity.com/paper/2302.03494/) (2023)
- [Software Engineering for AI-Based Systems: A Survey](https://mlsanity.com/paper/2105.01984/) (2021)
- [On the Potential of Artificial Intelligence Chatbots for Data Exploration of Federated Bioinformatics Knowledge Graphs](https://mlsanity.com/paper/2304.10427/) (2023)

## Built on by (most cited first, 6 of 6)

- [The Future of AI-Driven Software Engineering](https://mlsanity.com/paper/2406.07737/) (2024)
- [Metamorphic Testing of Large Language Models for Natural Language Processing](https://mlsanity.com/paper/2511.02108/) (2025)
- [System Test Case Design from Requirements Specifications: Insights and Challenges of Using ChatGPT](https://mlsanity.com/paper/2412.03693/) (2024)
- [MR-Coupler: Automated Metamorphic Test Generation via Functional Coupling Analysis](https://mlsanity.com/paper/2604.10126/) (2026)
- [Multi-Agent Specification-based Metamorphic Testing of FMU-Based Simulations](https://mlsanity.com/paper/2605.25101/) (2026)
- [Multi-Agent LLM-based Metamorphic Testing for REST APIs](https://mlsanity.com/paper/2605.28321/) (2026)

## Abstract (arXiv)

While ChatGPT is a well-known artificial intelligence chatbot being used to answer human's questions, one may want to discover its potential in advancing software testing. We examine the capability of ChatGPT in advancing the intelligence of software testing through a case study on metamorphic testing (MT), a state-of-the-art software testing technique. We ask ChatGPT to generate candidates of metamorphic relations (MRs), which are basically necessary properties of the object program and which traditionally require human intelligence to identify. These MR candidates are then evaluated in terms of correctness by domain experts. We show that ChatGPT can be used to generate new correct MRs to test several software systems. Having said that, the majority of MR candidates are either defined vaguely or incorrect, especially for systems that have never been tested with MT. ChatGPT can be used to advance software testing intelligence by proposing MR candidates that can be later adopted for implementing tests; but human intelligence should still inevitably be involved to justify and rectify their correctness.