# Unlocking UML Class Diagram Understanding in Vision Language Models

arXiv: https://arxiv.org/abs/2605.11634 · HTML: https://mlsanity.com/paper/2605.11634/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 15 of its 20 references that are in the map.
- First posted on arXiv 12 May 2026.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 15 of 15)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answering](https://mlsanity.com/paper/2504.05506/) (2025)
- [Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and Robustness](https://mlsanity.com/paper/2407.11229/) (2024)
- [LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact Language Model](https://mlsanity.com/paper/2404.01331/) (2024)
- [From Image to UML: First Results of Image Based UML Diagram Generation Using LLMs](https://mlsanity.com/paper/2404.11376/) (2024)
- [Limits and Gains of Test-Time Scaling in Vision-Language Reasoning](https://mlsanity.com/paper/2512.11109/) (2025)
- [MM-GEN: Enhancing Task Performance Through Targeted Multimodal Data Curation](https://mlsanity.com/paper/2501.04155/) (2025)
- [Structured Extraction from Business Process Diagrams Using Vision-Language Models](https://mlsanity.com/paper/2511.22448/) (2025)
- [VLM@school -- Evaluation of AI image understanding on German middle school knowledge](https://mlsanity.com/paper/2506.11604/) (2025)
- [NOMAD: A Multi-Agent LLM System for UML Class Diagram Generation from Natural Language Requirements](https://mlsanity.com/paper/2511.22409/) (2025)
- [Measuring Visual Understanding in Telecom domain: Performance Metrics for Image-to-UML conversion using VLMs](https://mlsanity.com/paper/2509.11667/) (2025)

## Abstract (arXiv)

Although Vision Language Models (VLMs) have seen tremendous progress across all kinds of use cases, they still fall behind in answering questions regard-ing diagrams compared to photos. Although progress has been made in the area of bar charts, line charts and other diagrams like that there is still few research concerned with other types of diagrams, e.g. in the computer science domain. Our work presents a benchmark for visual question answering based on UML class diagrams which is both challenging and manageable. We further construct a large-scale training dataset with 16.000 image-question-answer triples and show that a LoRA-based finetune easily outperforms Qwen 3.5 27B, which is a recent and well-performing VLM in many other benchmarks.