# OMPGPT: A Generative Pre-trained Transformer Model for OpenMP

arXiv: https://arxiv.org/abs/2401.16445 · HTML: https://mlsanity.com/paper/2401.16445/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 39 references that are in the map.
- First posted on arXiv 28 Jan 2024.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [The Stack: 3 TB of permissively licensed source code](https://mlsanity.com/paper/2211.15533/) (2022)
- [Structured Chain-of-Thought Prompting for Code Generation](https://mlsanity.com/paper/2305.06599/) (2023)
- [Prompting Is All You Need: Automated Android Bug Replay with Large Language Models](https://mlsanity.com/paper/2306.01987/) (2023)
- [Comparing Llama-2 and GPT-3 LLMs for HPC kernels generation](https://mlsanity.com/paper/2309.07103/) (2023)
- [Improving ChatGPT Prompt for Code Generation](https://mlsanity.com/paper/2305.08360/) (2023)
- [HPC-GPT: Integrating Large Language Model for High-Performance Computing](https://mlsanity.com/paper/2311.12833/) (2023)
- [LM4HPC: Towards Effective Language Model Application in High-Performance Computing](https://mlsanity.com/paper/2306.14979/) (2023)
- [Learning to Parallelize with OpenMP by Augmented Heterogeneous AST Representation](https://mlsanity.com/paper/2305.05779/) (2023)
- [MonoCoder: Domain-Specific Code Language Model for HPC Codes and Tasks](https://mlsanity.com/paper/2312.13322/) (2023)
- [HPC-Coder: Modeling Parallel Programs using Large Language Models](https://mlsanity.com/paper/2306.17281/) (2023)
- [Data Race Detection Using Large Language Models](https://mlsanity.com/paper/2308.07505/) (2023)
- [Advising OpenMP Parallelization via a Graph-Based Approach with Transformers](https://mlsanity.com/paper/2305.11999/) (2023)
- [Quantifying OpenMP: Statistical Insights into Usage and Adoption](https://mlsanity.com/paper/2308.08002/) (2023)
- [Performance Optimization using Multimodal Modeling and Heterogeneous GNN](https://mlsanity.com/paper/2304.12568/) (2023)
- [CompCodeVet: A Compiler-guided Validation and Enhancement Approach for Code Dataset](https://mlsanity.com/paper/2311.06505/) (2023)

## Built on by (most cited first, 16 of 16)

- [Performance-Aligned LLMs for Generating Fast Code](https://mlsanity.com/paper/2404.18864/) (2024)
- [HPC-Coder-V2: Studying Code LLMs Across Low-Resource Parallel Languages](https://mlsanity.com/paper/2412.15178/) (2024)
- [MPIrigen: MPI Code Generation through Domain-Specific Language Models](https://mlsanity.com/paper/2402.09126/) (2024)
- [OMPar: Automatic Parallelization with AI-Driven Source-to-Source Compilation](https://mlsanity.com/paper/2409.14771/) (2024)
- [Credit Risk Identification in Supply Chains Using Generative Adversarial Networks](https://mlsanity.com/paper/2501.10348/) (2025)
- [A Structure-Aware Framework for Learning Device Placements on Computation Graphs](https://mlsanity.com/paper/2405.14185/) (2024)
- [ChronoLLM: Customizing Language Models for Physics-Based Simulation Code Generation](https://mlsanity.com/paper/2508.13975/) (2025)
- [VibeCodeHPC: An Agent-Based Iterative Prompting Auto-Tuner for HPC Code Generation Using LLMs](https://mlsanity.com/paper/2510.00031/) (2025)
- [AI Assistants to Enhance and Exploit the PETSc Knowledge Base](https://mlsanity.com/paper/2506.20608/) (2025)
- [P4OMP: Retrieval-Augmented Prompting for OpenMP Parallelism in Serial Code](https://mlsanity.com/paper/2506.22703/) (2025)
- [An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSc](https://mlsanity.com/paper/2603.15976/) (2026)
- [Accelerating Latency-Critical Applications with AI-Powered Semi-Automatic Fine-Grained Parallelization on SMT Processors](https://mlsanity.com/paper/2509.00883/) (2025)
- [ACCeLLiuM: Supervised Fine-Tuning for Automated OpenACC Pragma Generation](https://mlsanity.com/paper/2509.20380/) (2025)
- [Cross-Model Cross-Language AI Coding Agent Performance: Accuracy and Speed of Parallel CLRS Algorithms](https://mlsanity.com/paper/2607.26083/) (2026)
- [HARGO: Heterogeneity-Aware Reward-Guided Optimization for RL Post-Training of LLMs on HPC Tasks](https://mlsanity.com/paper/2607.28301/) (2026)
- [Effect of Abstractions and Prompting Strategies on LLM-Guided High-Performance Optimizations](https://mlsanity.com/paper/2608.08085/) (2026)

## Abstract (arXiv)

Large language models (LLMs)such as ChatGPT have significantly advanced the field of Natural Language Processing (NLP). This trend led to the development of code-based large language models such as StarCoder, WizardCoder, and CodeLlama, which are trained extensively on vast repositories of code and programming languages. While the generic abilities of these code LLMs are useful for many programmers in tasks like code generation, the area of high-performance computing (HPC) has a narrower set of requirements that make a smaller and more domain-specific model a smarter choice. This paper presents OMPGPT, a novel domain-specific model meticulously designed to harness the inherent strengths of language models for OpenMP pragma generation. Furthermore, we leverage prompt engineering techniques from the NLP domain to create Chain-of-OMP, an innovative strategy designed to enhance OMPGPT's effectiveness. Our extensive evaluations demonstrate that OMPGPT outperforms existing large language models specialized in OpenMP tasks and maintains a notably smaller size, aligning it more closely with the typical hardware constraints of HPC environments. We consider our contribution as a pivotal bridge, connecting the advantage of language models with the specific demands of HPC tasks.