# On Codex Prompt Engineering for OCL Generation: An Empirical Study

arXiv: https://arxiv.org/abs/2303.16244 · HTML: https://mlsanity.com/paper/2303.16244/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 29 references that are in the map.
- First posted on arXiv 28 Mar 2023.

## Builds on (most cited first, 9 of 9)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [Synchromesh: Reliable code generation from pre-trained language models](https://mlsanity.com/paper/2201.11227/) (2022)
- [BashExplainer: Retrieval-Augmented Bash Code Comment Generation based on Fine-tuned CodeBERT](https://mlsanity.com/paper/2206.13325/) (2022)

## Built on by (most cited first, 8 of 8)

- [Agent READMEs: An Empirical Study of Context Files for Agentic Coding](https://mlsanity.com/paper/2511.12884/) (2025)
- [Exploring Prompt Engineering: A Systematic Review with SWOT Analysis](https://mlsanity.com/paper/2410.12843/) (2024)
- [PathOCL: Path-Based Prompt Augmentation for OCL Generation with GPT-4](https://mlsanity.com/paper/2405.12450/) (2024)
- [Do Prompt Patterns Affect Code Quality? A First Empirical Assessment of ChatGPT-Generated Code](https://mlsanity.com/paper/2504.13656/) (2025)
- [Optimizing Retrieval Augmented Generation for Object Constraint Language](https://mlsanity.com/paper/2505.13129/) (2025)
- [SHERPA: A Model-Driven Framework for Large Language Model Execution](https://mlsanity.com/paper/2509.00272/) (2025)
- [Leveraging LLMs to support co-evolution between definitions and instances of textual DSLs: A Systematic Evaluation](https://mlsanity.com/paper/2602.11904/) (2026)
- [Unify and Triumph: Polyglot, Diverse, and Self-Consistent Generation of Unit Tests with LLMs](https://mlsanity.com/paper/2503.16144/) (2025)

## Abstract (arXiv)

The Object Constraint Language (OCL) is a declarative language that adds constraints and object query expressions to MOF models. Despite its potential to provide precision and conciseness to UML models, the unfamiliar syntax of OCL has hindered its adoption. Recent advancements in LLMs, such as GPT-3, have shown their capability in many NLP tasks, including semantic parsing and text generation. Codex, a GPT-3 descendant, has been fine-tuned on publicly available code from GitHub and can generate code in many programming languages. We investigate the reliability of OCL constraints generated by Codex from natural language specifications. To achieve this, we compiled a dataset of 15 UML models and 168 specifications and crafted a prompt template with slots to populate with UML information and the target task, using both zero- and few-shot learning methods. By measuring the syntactic validity and execution accuracy metrics of the generated OCL constraints, we found that enriching the prompts with UML information and enabling few-shot learning increases the reliability of the generated OCL constraints. Furthermore, the results reveal a close similarity based on sentence embedding between the generated OCL constraints and the human-written ones in the ground truth, implying a level of clarity and understandability in the generated OCL constraints by Codex.