# From Image to UML: First Results of Image Based UML Diagram Generation Using LLMs

arXiv: https://arxiv.org/abs/2404.11376 · HTML: https://mlsanity.com/paper/2404.11376/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Apr 2024.

## Built on by (most cited first, 11 of 11)

- [Generative AI and Empirical Software Engineering: A Paradigm Shift](https://mlsanity.com/paper/2502.08108/) (2025)
- [Unified Modeling Language Code Generation from Diagram Images Using Multimodal Large Language Models](https://mlsanity.com/paper/2503.12293/) (2025)
- [Multi-modal Summarization in Model-Based Engineering: Automotive Software Development Case Study](https://mlsanity.com/paper/2503.04506/) (2025)
- [Assessing LLMs for Front-end Software Architecture Knowledge](https://mlsanity.com/paper/2502.19518/) (2025)
- [The importance of visual modelling languages in generative software engineering](https://mlsanity.com/paper/2411.17976/) (2024)
- [Measuring Visual Understanding in Telecom domain: Performance Metrics for Image-to-UML conversion using VLMs](https://mlsanity.com/paper/2509.11667/) (2025)
- [Code2UML: Agentic LLMs with context engineering for scalable software visualization](https://mlsanity.com/paper/2605.24453/) (2026)
- [LLM-based Automated Architecture View Generation: Where Are We Now?](https://mlsanity.com/paper/2603.21178/) (2026)
- [Unlocking UML Class Diagram Understanding in Vision Language Models](https://mlsanity.com/paper/2605.11634/) (2026)
- [Prior Bias in Vision Language Models on UML Diagram Interpretation](https://mlsanity.com/paper/2607.02853/) (2026)
- [CAS2UML: A Handwritten Sketch-to-PlantUML Dataset for Class and Activity Diagrams](https://mlsanity.com/paper/2608.07036/) (2026)

## Abstract (arXiv)

In software engineering processes, systems are first specified using a modeling language such as UML. These initial designs are often collaboratively created, many times in meetings where different domain experts use whiteboards, paper or other types of quick supports to create drawings and blueprints that then will need to be formalized. These proper, machine-readable, models are key to ensure models can be part of automated processes (e.g. input of a low-code generation pipeline, a model-based testing system, ...). But going from hand-drawn diagrams to actual models is a time-consuming process that sometimes ends up with such drawings just added as informal images to the software documentation, reducing their value a lot. To avoid this tedious task, we explore the usage of Large Language Models (LLM) to generate the formal representation of (UML) models from a given drawing. More specifically, we have evaluated the capabilities of different LLMs to convert images of UML class diagrams into the actual models represented in the images. While the results are good enough to use such an approach as part of a model-driven engineering pipeline we also highlight some of their current limitations and the need to keep the human in the loop to overcome those limitations.