Leveraging LLMs to support co-evolution between definitions and instances of textual DSLs: A Systematic Evaluation
Not on the map. Why?
Abstract (arXiv)
Software languages evolve over time for reasons such as feature additions. When grammars evolve, textual instances that originally conformed to them may become outdated. While model-driven engineering provides many techniques for co-evolving models with metamodel changes, these approaches are not designed for textual DSLs and may lose human-relevant information such as layout and comments. This study systematically evaluates the potential of large language models (LLMs) for co-evolving grammars and instances of textual DSLs. Using Claude Sonnet 4.5 and GPT-5.2 across ten case languages with ten runs each, we assess both correctness and preservation of human-oriented information. Results show strong performance on small-scale cases (≥94% precision and recall for instances requiring fewer than 20 modified lines), but performance degraded with scale: Claude maintains 85% recall at 40 lines, while GPT fails on the largest instances. Response time increases substantially with instance size, and grammar evolution complexity and deletion granularity affect performance more than change type. These findings clarify when LLM-based co-evolution is effective and where current limitations remain.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 17 of its 75 references that are in the map. First posted on arXiv 12 Feb 2026.
Builds on
17 of 75 references.
References · 17
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
- 2023Using an LLM to Help With Code Understandingcited by 228
- 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 122
- 2024A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languagescited by 66
- 2023Enhancing Zero-Shot Chain-of-Thought Reasoning in Large Language Models through Logiccited by 35
- 2022Towards using Few-Shot Prompt Learning for Automating Model Completioncited by 13
- 2024On the use of Large Language Models in Model-Driven Engineeringcited by 13
- 2023Prompt Engineering or Fine-Tuning: An Empirical Assessment of LLMs for Codecited by 9
- 2023On Codex Prompt Engineering for OCL Generation: An Empirical Studycited by 8
5 more
- 2021Recommending Metamodel Concepts during Modeling Activities with Pre-Trained Language Modelscited by 7
- 2025Optimizing Retrieval Augmented Generation for Object Constraint Languagecited by 4
- 2025Development and Evolution of Xtext-based DSLs on GitHub: An Empirical Investigationcited by 3
- 2025Leveraging LLMs to support co-evolution between definitions and instances of textual DSLscited by 3
- 2020On the Evolution of Programming Languagescited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.