# (How) Do Large Language Models Understand High-Level Message Sequence Charts?

arXiv: https://arxiv.org/abs/2605.13773 · HTML: https://mlsanity.com/paper/2605.13773/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 25 references that are in the map.
- First posted on arXiv 13 May 2026.

## Builds on (most cited first, 12 of 12)

- [EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checking](https://mlsanity.com/paper/2502.12466/) (2025)
- [Exploring Code Analysis: Zero-Shot Insights on Syntax and Semantics with LLMs](https://mlsanity.com/paper/2305.12138/) (2023)
- [Do Large Code Models Understand Programming Concepts? Counterfactual Analysis for Code Predicates](https://mlsanity.com/paper/2402.05980/) (2024)
- [Software Architecture Meets LLMs: A Systematic Literature Review](https://mlsanity.com/paper/2505.16697/) (2025)
- [The Code Barrier: What LLMs Actually Understand?](https://mlsanity.com/paper/2504.10557/) (2025)
- [Using LLMs in Generating Design Rationale for Software Architecture Decisions](https://mlsanity.com/paper/2504.20781/) (2025)
- [Comparative Analysis of Carbon Footprint in Manual vs. LLM-Assisted Code Development](https://mlsanity.com/paper/2505.04521/) (2025)
- [The Future of Generative AI in Software Engineering: A Vision from Industry and Academia in the European GENIUS Project](https://mlsanity.com/paper/2511.01348/) (2025)
- [Assessing LLMs for Front-end Software Architecture Knowledge](https://mlsanity.com/paper/2502.19518/) (2025)
- [Benchmarking and Evaluating VLMs for Software Architecture Diagram Understanding](https://mlsanity.com/paper/2604.04009/) (2026)
- [How Robustly do LLMs Understand Execution Semantics?](https://mlsanity.com/paper/2604.16320/) (2026)
- [Using LLMs to Evaluate Architecture Documents: Results from a Digital Marketplace Environment](https://mlsanity.com/paper/2601.19693/) (2026)

## Built on by (most cited first, 1 of 1)

- [CordisBench: Can Language Models Reason About Component Lifecycles in Dynamic Agent Harnesses?](https://mlsanity.com/paper/2609.01600/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are being employed widely to automate tasks across the software development life-cycle. It is, however, unclear whether these tasks are performed consistently with respect to the semantics of the artefacts being handled. This question is particularly under-researched concerning architectural design specification. In this paper, we address this question for High-Level Message Sequence Charts (HMSCs). These are visual models with a rigorous formal semantics that have been used for various purposes, including as a foundation for Sequence Diagrams in the Unified Modelling Language (UML). We examine whether LLMs "understand" the semantics of HMSCs by examining three LLMs (Gemini-3, GPT-5.4, and Qwen-3.6) on how they perform 129 semantic tasks ranging from querying basic semantic constructs in HMSCs (i.e., events and their ordering) to semantic-preserving abstractions and compositions, and calculating the set of traces and trace-equivalent labelled transition systems. The results show that LLMs only have a modest understanding of the formal semantics of HMSCs (ca. 52% overall accuracy), with great variability across different semantic concepts: while LLMs seem to understand the basic semantic concepts of MSCs (ca. 88% accuracy), they struggle with semantic reasoning in tasks involving abstraction and composition (ca. 36% accuracy) and traces and LTSs (ca. 42% accuracy). In particular, all three LLMs struggle with the notions of co-region and explicit causal dependencies and never employed them in semantic-preserving transformations.