# Beyond Code Generation: Assessing Code LLM Maturity with Postconditions

arXiv: https://arxiv.org/abs/2407.14118 · HTML: https://mlsanity.com/paper/2407.14118/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jul 2024.

## Built on by (most cited first, 4 of 4)

- [EyeMulator: Improving Code Language Models by Mimicking Human Visual Attention](https://mlsanity.com/paper/2508.16771/) (2025)
- [Spec-Harness: Assessing and Guiding LLM-Synthesized Formal Specifications Beyond Verifiability](https://mlsanity.com/paper/2604.00280/) (2026)
- [Prompt engineering and framework: implementation to increase code reliability based guideline for LLMs](https://mlsanity.com/paper/2506.10989/) (2025)
- [FVRuleLearner: Operator-Level Reasoning Tree (Op-Tree)-Based Rules Learning for Formal Verification](https://mlsanity.com/paper/2604.03245/) (2026)

## Abstract (arXiv)

Most existing code Large Language Model (LLM) benchmarks, e.g., EvalPlus, focus on the code generation tasks. Namely, they contain a natural language description of a problem and ask the LLM to write code to solve the problem. We argue that they do not capture all capabilities needed to assess the quality of a code LLM. In this paper, we propose a code LLM maturity model, based on the postcondition generation problem, to access a more complete set of code LLM capabilities. We choose the postcondition generation problem as it requires the code LLM to understand the code including semantics, natural language, and also have the capability to generate unambiguous postconditions in programming languages (i.e., the generation capablity). Moreover, postconditions have various types, requiring different levels of these capabilities, making it suitable to evaluate the maturity of the code LLM. Based on our designed maturity model, we augment the EvalPlus dataset to a postcondition testing benchmark, and evaluated several open-sourced models. Our results highlight the necessary improvements needed for better LLMs for code. Code: https://github.com/MatureModel/PostcondGen