# MermaidSeqBench: An Evaluation Benchmark for NL-to-Mermaid Sequence Diagram Generation

arXiv: https://arxiv.org/abs/2511.14967 · HTML: https://mlsanity.com/paper/2511.14967/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 35 references that are in the map.
- First posted on arXiv 18 Nov 2025.

## Builds on (most cited first, 11 of 11)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges](https://mlsanity.com/paper/2406.12624/) (2024)
- [A Comparative Analysis of Large Language Models for Code Documentation Generation](https://mlsanity.com/paper/2312.10349/) (2023)
- [Model Generation with LLMs: From Requirements to UML Sequence Diagrams](https://mlsanity.com/paper/2404.06371/) (2024)
- [Conversational AI as a Coding Assistant: Understanding Programmers' Interactions with and Expectations from Large Language Models for Coding](https://mlsanity.com/paper/2503.16508/) (2025)
- [Free and Customizable Code Documentation with LLMs: A Fine-Tuning Approach](https://mlsanity.com/paper/2412.00726/) (2024)
- [How LLMs Aid in UML Modeling: An Exploratory Study with Novice Analysts](https://mlsanity.com/paper/2404.17739/) (2024)
- [Behavioral Augmentation of UML Class Diagrams: An Empirical Study of Large Language Models for Method Generation](https://mlsanity.com/paper/2506.00788/) (2025)

## Built on by (most cited first, 3 of 3)

- [Evaluating Ill-Defined Tasks in Large Language Models](https://mlsanity.com/paper/2603.17067/) (2026)
- [Math-Vision Diagrams: A Comprehensive Benchmark for Evaluating LLM Mathematical Diagram Generation Capabilities](https://mlsanity.com/paper/2608.08964/) (2026)
- [A Systematic Approach for Large Language Models Debugging](https://mlsanity.com/paper/2604.23027/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown great promise in generating structured diagrams from natural language descriptions, particularly Mermaid sequence diagrams for software engineering. However, the lack of existing benchmarks to assess the LLM's correctness on this task hinders rigorous, systematic evaluation and principled comparison of model capabilities on this task. To address this shortcoming, we introduce MermaidSeqBench, a human-verified and synthetically extended benchmark for assessing LLM capabilities in generating Mermaid sequence diagrams from natural language prompts. The benchmark consists of 132 samples developed via a hybrid methodology of human-verified flows, LLM-based augmentation, and rule-based expansion. The evaluation uses an LLM-as-a-judge model to assess generation across various fine-grained metrics such as syntax correctness, activation handling, error handling, and practical usability. To demonstrate the effectiveness and flexibility of our benchmark, we perform initial evaluations on numerous state-of-the-art LLMs with multiple LLM judges which reveal significant capability gaps across models and evaluation modes. MermaidSeqBench provides a foundation for evaluating structured diagram generation and advances the scientific understanding of LLM capabilities and limitations in structured generation tasks.