# How Effective are Large Language Models in Generating Software Specifications?

arXiv: https://arxiv.org/abs/2306.03324 · HTML: https://mlsanity.com/paper/2306.03324/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jun 2023.

## Built on by (most cited first, 12 of 12)

- [MARE: Multi-Agents Collaboration Framework for Requirements Engineering](https://mlsanity.com/paper/2405.03256/) (2024)
- [AIReg-Bench: Benchmarking Language Models That Assess AI Regulation Compliance](https://mlsanity.com/paper/2510.01474/) (2025)
- [ConDefects: A New Dataset to Address the Data Leakage Concern for LLM-based Fault Localization and Program Repair](https://mlsanity.com/paper/2310.16253/) (2023)
- [Using LLMs in Generating Design Rationale for Software Architecture Decisions](https://mlsanity.com/paper/2504.20781/) (2025)
- [Agentic Specification Generator for Move Programs](https://mlsanity.com/paper/2509.24515/) (2025)
- [SysMoBench: Evaluating AI on Formally Modeling Complex Real-World Systems](https://mlsanity.com/paper/2509.23130/) (2025)
- [Prompting for Performance: Exploring LLMs for Configuring Software](https://mlsanity.com/paper/2507.09790/) (2025)
- [Reversa: A Reverse Documentation Engineering Framework for Converting Legacy Software into Operational Specifications for AI Agents](https://mlsanity.com/paper/2605.18684/) (2026)
- [Improving Performance of Commercially Available AI Products in a Multi-Agent Configuration](https://mlsanity.com/paper/2410.22129/) (2024)
- [ReqGenX: An Empirical Study of Atomic Decomposition, Artifact Regeneration, and Reconstruction for Legacy SRS Documents](https://mlsanity.com/paper/2607.16564/) (2026)
- [How Well Can AI Generate Backlogs from App Mockups?](https://mlsanity.com/paper/2607.22902/) (2026)
- [On Synthesis of Metric Interval Temporal Logics](https://mlsanity.com/paper/2609.01032/) (2026)

## Abstract (arXiv)

Software specifications are essential for many Software Engineering (SE) tasks such as bug detection and test generation. Many existing approaches are proposed to extract the specifications defined in natural language form (e.g., comments) into formal machine readable form (e.g., first order logic). However, existing approaches suffer from limited generalizability and require manual efforts. The recent emergence of Large Language Models (LLMs), which have been successfully applied to numerous SE tasks, offers a promising avenue for automating this process. In this paper, we conduct the first empirical study to evaluate the capabilities of LLMs for generating software specifications from software comments or documentation. We evaluate LLMs performance with Few Shot Learning (FSL) and compare the performance of 13 state of the art LLMs with traditional approaches on three public datasets. In addition, we conduct a comparative diagnosis of the failure cases from both LLMs and traditional methods, identifying their unique strengths and weaknesses. Our study offers valuable insights for future research to improve specification generation.