# The importance of visual modelling languages in generative software engineering

arXiv: https://arxiv.org/abs/2411.17976 · HTML: https://mlsanity.com/paper/2411.17976/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 27 Nov 2024.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [PAL: Program-aided Language Models](https://mlsanity.com/paper/2211.10435/) (2022)
- [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.06211/) (2024)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [Generated Knowledge Prompting for Commonsense Reasoning](https://mlsanity.com/paper/2110.08387/) (2021)
- [ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation](https://mlsanity.com/paper/2308.01861/) (2023)
- [Guiding Large Language Models via Directional Stimulus Prompting](https://mlsanity.com/paper/2302.11520/) (2023)
- [Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)](https://mlsanity.com/paper/2304.06815/) (2023)
- [ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design](https://mlsanity.com/paper/2303.07839/) (2023)
- [Advancing Requirements Engineering through Generative AI: Assessing the Role of LLMs](https://mlsanity.com/paper/2310.13976/) (2023)
- [Is AI the better programming partner? Human-Human Pair Programming vs. Human-AI pAIr Programming](https://mlsanity.com/paper/2306.05153/) (2023)
- [Experimenting a New Programming Practice with LLMs](https://mlsanity.com/paper/2401.01062/) (2024)
- [Large Language Models Based Automatic Synthesis of Software Specifications](https://mlsanity.com/paper/2304.09181/) (2023)
- [From Image to UML: First Results of Image Based UML Diagram Generation Using LLMs](https://mlsanity.com/paper/2404.11376/) (2024)
- [How LLMs Aid in UML Modeling: An Exploratory Study with Novice Analysts](https://mlsanity.com/paper/2404.17739/) (2024)
- [Sketches and Diagrams in Practice](https://mlsanity.com/paper/1706.09172/) (2017)

## Built on by (most cited first, 1 of 1)

- [Multi-modal Summarization in Model-Based Engineering: Automotive Software Development Case Study](https://mlsanity.com/paper/2503.04506/) (2025)

## Abstract (arXiv)

Multimodal GPTs represent a watershed in the interplay between Software Engineering and Generative Artificial Intelligence. GPT-4 accepts image and text inputs, rather than simply natural language. We investigate relevant use cases stemming from these enhanced capabilities of GPT-4. To the best of our knowledge, no other work has investigated similar use cases involving Software Engineering tasks carried out via multimodal GPTs prompted with a mix of diagrams and natural language.