# METAMON: Finding Inconsistencies between Program Documentation and Behavior using Metamorphic LLM Queries

arXiv: https://arxiv.org/abs/2502.02794 · HTML: https://mlsanity.com/paper/2502.02794/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Feb 2025.

## Built on by (most cited first, 3 of 3)

- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [CodeFuse-CommitEval: Towards Benchmarking LLM's Power on Commit Message and Code Change Inconsistency Detection](https://mlsanity.com/paper/2511.19875/) (2025)
- [Multi-Large Language Model Orchestrated Severity Assessment of Clinical Records (MOSAIC)](https://mlsanity.com/paper/2607.05032/) (2026)

## Abstract (arXiv)

Code documentation can, if written precisely, help developers better understand the code they accompany. However, unlike code, code documentation cannot be automatically verified via execution, potentially leading to inconsistencies between documentation and the actual behavior. While such inconsistencies can be harmful for the developer's understanding of the code, checking and finding them remains a costly task due to the involvement of human engineers. This paper proposes METAMON, which uses an existing search-based test generation technique to capture the current program behavior in the form of test cases, and subsequently uses LLM-based code reasoning to identify the generated regression test oracles that are not consistent with the program specifications in the documentation. METAMON is supported in this task by metamorphic testing and self-consistency. An empirical evaluation against 9,482 pairs of code documentation and code snippets, generated using five open-source projects from Defects4J v2.0.1, shows that METAMON can classify the code-and-documentation inconsistencies with a precision of 0.72 and a recall of 0.48.