Automated Repair of Declarative Software Specifications in the Era of Large Language Models
Not on the map. Why?
Abstract (arXiv)
The growing adoption of declarative software specification languages, coupled with their inherent difficulty in debugging, has underscored the need for effective and automated repair techniques applicable to such languages. Researchers have recently explored various methods to automatically repair declarative software specifications, such as template-based repair, feedback-driven iterative repair, and bounded exhaustive approaches. The latest developments in large language models provide new opportunities for the automatic repair of declarative specifications. In this study, we assess the effectiveness of utilizing OpenAI's ChatGPT to repair software specifications written in the Alloy declarative language. Unlike imperative languages, specifications in Alloy are not executed but rather translated into logical formulas and evaluated using backend constraint solvers to identify specification instances and counterexamples to assertions. Our evaluation focuses on ChatGPT's ability to improve the correctness and completeness of Alloy declarative specifications through automatic repairs. We analyze the results produced by ChatGPT and compare them with those of leading automatic Alloy repair methods. Our study revealed that while ChatGPT falls short in comparison to existing techniques, it was able to successfully repair bugs that no other technique could address. Our analysis also identified errors in ChatGPT's generated repairs, including improper operator usage, type errors, higher-order logic misuse, and relational arity mismatches. Additionally, we observed instances of hallucinations in ChatGPT-generated repairs and inconsistency in its results. Our study provides valuable insights for software practitioners, researchers, and tool builders considering ChatGPT for declarative specification repairs.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 88 references that are in the map. First posted on arXiv 19 Oct 2023.
Builds on
22 of 88 references.
References · 22
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
- 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
- 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
- 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
10 more
- 2022A Systematic Evaluation of Large Language Models of Codecited by 360
- 2015Semi-supervised Sequence Learningcited by 198
- 2023Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPTcited by 125
- 2023ChatGPT for Programming Numerical Methodscited by 26
- 2018FixMiner: Mining Relevant Fix Patterns for Automated Program Repaircited by 22
- 2021Teaching Autoregressive Language Models Complex Tasks By Demonstrationcited by 15
- 2019iFixR: Bug Report driven Program Repaircited by 11
- 2018Fault Localization for Declarative Models in Alloycited by 1
- 2019Connecting Program Synthesis and Reachability: Automatic Program Repair using Test-Input Generationcited by 1
- 2021Bounded Exhaustive Search of Alloy Specification Repairscited by 1
Built on by
Citing papers per year: 2025: 1.
1 paper.