# Recommending Metamodel Concepts during Modeling Activities with Pre-Trained Language Models

arXiv: https://arxiv.org/abs/2104.01642 · HTML: https://mlsanity.com/paper/2104.01642/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 50 references that are in the map.
- First posted on arXiv 4 Apr 2021.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code](https://mlsanity.com/paper/2003.07914/) (2020)
- [Fast and Memory-Efficient Neural Code Completion](https://mlsanity.com/paper/2004.13651/) (2020)
- [Learning and Evaluating Contextual Embedding of Source Code](https://mlsanity.com/paper/2001.00059/) (2019)
- [SCELMo: Source Code Embeddings from Language Models](https://mlsanity.com/paper/2004.13214/) (2020)
- [MAR: A structure-based search engine for models](https://mlsanity.com/paper/2008.11858/) (2020)

## Built on by (most cited first, 7 of 7)

- [Prompting or Fine-tuning? A Comparative Study of Large Language Models for Taxonomy Construction](https://mlsanity.com/paper/2309.01715/) (2023)
- [Towards using Few-Shot Prompt Learning for Automating Model Completion](https://mlsanity.com/paper/2212.03404/) (2022)
- [On the Utility of Domain Modeling Assistance with Large Language Models](https://mlsanity.com/paper/2410.12577/) (2024)
- [Software Model Evolution with Large Language Models: Experiments on Simulated, Public, and Industrial Datasets](https://mlsanity.com/paper/2406.17651/) (2024)
- [Behavioral Augmentation of UML Class Diagrams: An Empirical Study of Large Language Models for Method Generation](https://mlsanity.com/paper/2506.00788/) (2025)
- [Leveraging LLMs to support co-evolution between definitions and instances of textual DSLs: A Systematic Evaluation](https://mlsanity.com/paper/2602.11904/) (2026)
- [Towards Automatic Support of Software Model Evolution with Large Language~Models](https://mlsanity.com/paper/2312.12404/) (2023)

## Abstract (arXiv)

The design of conceptually sound metamodels that embody proper semantics in relation to the application domain is particularly tedious in Model-Driven Engineering. As metamodels define complex relationships between domain concepts, it is crucial for a modeler to define these concepts thoroughly while being consistent with respect to the application domain. We propose an approach to assist a modeler in the design of a metamodel by recommending relevant domain concepts in several modeling scenarios. Our approach does not require to extract knowledge from the domain or to hand-design completion rules. Instead, we design a fully data-driven approach using a deep learning model that is able to abstract domain concepts by learning from both structural and lexical metamodel properties in a corpus of thousands of independent metamodels. We evaluate our approach on a test set containing 166 metamodels, unseen during the model training, with more than 5000 test samples. Our preliminary results show that the trained model is able to provide accurate top-5 lists of relevant recommendations for concept renaming scenarios. Although promising, the results are less compelling for the scenario of the iterative construction of the metamodel, in part because of the conservative strategy we use to evaluate the recommendations.