# Software Model Evolution with Large Language Models: Experiments on Simulated, Public, and Industrial Datasets

arXiv: https://arxiv.org/abs/2406.17651 · HTML: https://mlsanity.com/paper/2406.17651/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 79 references that are in the map.
- First posted on arXiv 25 Jun 2024.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [Few-shot training LLMs for project-specific code-summarization](https://mlsanity.com/paper/2207.04237/) (2022)
- [A Short Tutorial on The Weisfeiler-Lehman Test And Its Variants](https://mlsanity.com/paper/2201.07083/) (2022)
- [Large Language Models and Simple, Stupid Bugs](https://mlsanity.com/paper/2303.11455/) (2023)
- [No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence](https://mlsanity.com/paper/2207.11680/) (2022)
- [Towards Human-Bot Collaborative Software Architecting with ChatGPT](https://mlsanity.com/paper/2302.14600/) (2023)
- [Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programming](https://mlsanity.com/paper/2111.07875/) (2021)
- [An Empirical Study on the Usage of Transformer Models for Code Completion](https://mlsanity.com/paper/2108.01585/) (2021)
- [Towards using Few-Shot Prompt Learning for Automating Model Completion](https://mlsanity.com/paper/2212.03404/) (2022)
- [Recommending Metamodel Concepts during Modeling Activities with Pre-Trained Language Models](https://mlsanity.com/paper/2104.01642/) (2021)
- [Learning Domain-Specific Edit Operations from Model Repositories with Frequent Subgraph Mining](https://mlsanity.com/paper/2108.01001/) (2021)
- [Quality Guidelines for Research Artifacts in Model-Driven Engineering](https://mlsanity.com/paper/2108.04652/) (2021)
- [MemoRec: A Recommender System for Assisting Modelers in Specifying Metamodels](https://mlsanity.com/paper/2203.06068/) (2022)

## Built on by (most cited first, 1 of 1)

- [Rethinking Autonomy: Preventing Failures in AI-Driven Software Engineering](https://mlsanity.com/paper/2508.11824/) (2025)

## Abstract (arXiv)

Modeling structure and behavior of software systems plays a crucial role in the industrial practice of software engineering. As with other software engineering artifacts, software models are subject to evolution. Supporting modelers in evolving software models with recommendations for model completions is still an open problem, though. In this paper, we explore the potential of large language models for this task. In particular, we propose an approach, RAMC, leveraging large language models, model histories, and retrieval-augmented generation for model completion. Through experiments on three datasets, including an industrial application, one public open-source community dataset, and one controlled collection of simulated model repositories, we evaluate the potential of large language models for model completion with RAMC. We found that large language models are indeed a promising technology for supporting software model evolution (62.30% semantically correct completions on real-world industrial data and up to 86.19% type-correct completions). The general inference capabilities of large language models are particularly useful when dealing with concepts for which there are few, noisy, or no examples at all.