# An Evaluation of Requirements Modeling for Cyber-Physical Systems via LLMs

arXiv: https://arxiv.org/abs/2408.02450 · HTML: https://mlsanity.com/paper/2408.02450/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Aug 2024.

## Built on by (most cited first, 7 of 7)

- [From Informal to Formal -- Incorporating and Evaluating LLMs on Natural Language Requirements to Verifiable Formal Proofs](https://mlsanity.com/paper/2501.16207/) (2025)
- [Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agents](https://mlsanity.com/paper/2505.05283/) (2025)
- [MAAD: Automate Software Architecture Design through Knowledge-Driven Multi-Agent Collaboration](https://mlsanity.com/paper/2507.21382/) (2025)
- [Knowledge-Guided Multi-Agent Framework for Automated Requirements Development: A Vision](https://mlsanity.com/paper/2506.22656/) (2025)
- [Bridging Requirements and Architecture: Multi-Agent Orchestration with External Knowledge and Hierarchical Memory](https://mlsanity.com/paper/2606.01385/) (2026)
- [Relationally Guided Use Case Modeling with LLMs](https://mlsanity.com/paper/2609.18291/) (2026)
- [DTScribe: Generating Models for Digital Twin Reporting using Local Large Language Models](https://mlsanity.com/paper/2610.18635/) (2026)

## Abstract (arXiv)

Cyber-physical systems (CPSs) integrate cyber and physical components and enable them to interact with each other to meet user needs. The needs for CPSs span rich application domains such as healthcare and medicine, smart home, smart building, etc. This indicates that CPSs are all about solving real-world problems. With the increasing abundance of sensing devices and effectors, the problems wanted to solve with CPSs are becoming more and more complex. It is also becoming increasingly difficult to extract and express CPS requirements accurately. Problem frame approach aims to shape real-world problems by capturing the characteristics and interconnections of components, where the problem diagram is central to expressing the requirements. CPSs requirements are generally presented in domain-specific documents that are normally expressed in natural language. There is currently no effective way to extract problem diagrams from natural language documents. CPSs requirements extraction and modeling are generally done manually, which is time-consuming, labor-intensive, and error-prone. Large language models (LLMs) have shown excellent performance in natural language understanding. It can be interesting to explore the abilities of LLMs to understand domain-specific documents and identify modeling elements, which this paper is working on. To achieve this goal, we first formulate two tasks (i.e., entity recognition and interaction extraction) and propose a benchmark called CPSBench. Based on this benchmark, extensive experiments are conducted to evaluate the abilities and limitations of seven advanced LLMs. We find some interesting insights. Finally, we establish a taxonomy of LLMs hallucinations in CPSs requirements modeling using problem diagrams. These results will inspire research on the use of LLMs for automated CPSs requirements modeling.