# Learning to Parallelize in a Shared-Memory Environment with Transformers arXiv: https://arxiv.org/abs/2204.12835 ยท HTML: https://mlsanity.com/paper/2204.12835/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 25 of its 93 references that are in the map. - First posted on arXiv 27 Apr 2022. ## Builds on (most cited first, 25 of 25) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016) - [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014) - [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020) - [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019) - [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017) - [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015) - [Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1604.02201/) (2016) - [Retrieval Augmented Code Generation and Summarization](https://mlsanity.com/paper/2108.11601/) (2021) - [Improved Code Summarization via a Graph Neural Network](https://mlsanity.com/paper/2004.02843/) (2020) - [A Neural Model for Generating Natural Language Summaries of Program Subroutines](https://mlsanity.com/paper/1902.01954/) (2019) - [Low-resource Deep Entity Resolution with Transfer and Active Learning](https://mlsanity.com/paper/1906.08042/) (2019) - [Computational Sociolinguistics: A Survey](https://mlsanity.com/paper/1508.07544/) (2015) - [Improved Automatic Summarization of Subroutines via Attention to File Context](https://mlsanity.com/paper/2004.04881/) (2020) - [A Grammar-Based Structural CNN Decoder for Code Generation](https://mlsanity.com/paper/1811.06837/) (2018) - [A Survey on Machine Learning Techniques for Source Code Analysis](https://mlsanity.com/paper/2110.09610/) (2021) - [ProGraML: Graph-based Deep Learning for Program Optimization and Analysis](https://mlsanity.com/paper/2003.10536/) (2020) - [Exploiting Out-of-Domain Parallel Data through Multilingual Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1907.03060/) (2019) - [ComPar: Optimized Multi-Compiler for Automatic OpenMP S2S Parallelization](https://mlsanity.com/paper/2005.13304/) (2020) - [TreeCaps: Tree-Structured Capsule Networks for Program Source Code Processing](https://mlsanity.com/paper/1910.12306/) (2019) - [DeepSCC: Source Code Classification Based on Fine-Tuned RoBERTa](https://mlsanity.com/paper/2110.00914/) (2021) - [AutOMP: An Automatic OpenMP Parallelization Generator for Variable-Oriented High-Performance Scientific Codes](https://mlsanity.com/paper/1707.07137/) (2017) - [Machine Learning Framwork for Performance Anomaly in OpenMP Multi-Threaded Systems](https://mlsanity.com/paper/2011.02914/) (2020) ## Built on by (most cited first, 10 of 10) - [The Landscape and Challenges of HPC Research and LLMs](https://mlsanity.com/paper/2402.02018/) (2024) - [LM4HPC: Towards Effective Language Model Application in High-Performance Computing](https://mlsanity.com/paper/2306.14979/) (2023) - [Learning to Parallelize with OpenMP by Augmented Heterogeneous AST Representation](https://mlsanity.com/paper/2305.05779/) (2023) - [Scope is all you need: Transforming LLMs for HPC Code](https://mlsanity.com/paper/2308.09440/) (2023) - [MonoCoder: Domain-Specific Code Language Model for HPC Codes and Tasks](https://mlsanity.com/paper/2312.13322/) (2023) - [Advising OpenMP Parallelization via a Graph-Based Approach with Transformers](https://mlsanity.com/paper/2305.11999/) (2023) - [MPI-rical: Data-Driven MPI Distributed Parallelism Assistance with Transformers](https://mlsanity.com/paper/2305.09438/) (2023) - [MPIrigen: MPI Code Generation through Domain-Specific Language Models](https://mlsanity.com/paper/2402.09126/) (2024) - [OMPar: Automatic Parallelization with AI-Driven Source-to-Source Compilation](https://mlsanity.com/paper/2409.14771/) (2024) - [ParaCodex: A Profiling-Guided Autonomous Coding Agent for Reliable Parallel Code Generation and Translation](https://mlsanity.com/paper/2601.04327/) (2026) ## Abstract (arXiv) In past years, the world has switched to many-core and multi-core shared memory architectures. As a result, there is a growing need to utilize these architectures by introducing shared memory parallelization schemes to software applications. OpenMP is the most comprehensive API that implements such schemes, characterized by a readable interface. Nevertheless, introducing OpenMP into code is challenging due to pervasive pitfalls in management of parallel shared memory. To facilitate the performance of this task, many source-to-source (S2S) compilers have been created over the years, tasked with inserting OpenMP directives into code automatically. In addition to having limited robustness to their input format, these compilers still do not achieve satisfactory coverage and precision in locating parallelizable code and generating appropriate directives. In this work, we propose leveraging recent advances in ML techniques, specifically in natural language processing (NLP), to replace S2S compilers altogether. We create a database (corpus), Open-OMP, specifically for this goal. Open-OMP contains over 28,000 code snippets, half of which contain OpenMP directives while the other half do not need parallelization at all with high probability. We use the corpus to train systems to automatically classify code segments in need of parallelization, as well as suggest individual OpenMP clauses. We train several transformer models, named PragFormer, for these tasks, and show that they outperform statistically-trained baselines and automatic S2S parallelization compilers in both classifying the overall need for an OpenMP directive and the introduction of private and reduction clauses. Our source code and database are available at: https://github.com/Scientific-Computing-Lab-NRCN/PragFormer.