# Can Large Language Models Write Parallel Code?

arXiv: https://arxiv.org/abs/2401.12554 · HTML: https://mlsanity.com/paper/2401.12554/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 54 references that are in the map.
- First posted on arXiv 23 Jan 2024.

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [The Stack: 3 TB of permissively licensed source code](https://mlsanity.com/paper/2211.15533/) (2022)
- [VerilogEval: Evaluating Large Language Models for Verilog Code Generation](https://mlsanity.com/paper/2309.07544/) (2023)
- [SantaCoder: don't reach for the stars!](https://mlsanity.com/paper/2301.03988/) (2023)
- [ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation](https://mlsanity.com/paper/2308.01861/) (2023)
- [A Transformer-based Approach for Source Code Summarization](https://mlsanity.com/paper/2005.00653/) (2020)
- [Comparing Llama-2 and GPT-3 LLMs for HPC kernels generation](https://mlsanity.com/paper/2309.07103/) (2023)
- [BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models](https://mlsanity.com/paper/2308.16458/) (2023)
- [Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generation](https://mlsanity.com/paper/2306.15121/) (2023)
- [LLM4VV: Developing LLM-Driven Testsuite for Compiler Validation](https://mlsanity.com/paper/2310.04963/) (2023)
- [Learning to Reduce False Positives in Analytic Bug Detectors](https://mlsanity.com/paper/2203.09907/) (2022)
- [Scope is all you need: Transforming LLMs for HPC Code](https://mlsanity.com/paper/2308.09440/) (2023)
- [Learning code summarization from a small and local dataset](https://mlsanity.com/paper/2206.00804/) (2022)

## Built on by (most cited first, 30 of 36)

- [How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark](https://mlsanity.com/paper/2406.06647/) (2024)
- [LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks](https://mlsanity.com/paper/2502.06215/) (2025)
- [Geak: Introducing Triton Kernel AI Agent & Evaluation Benchmarks](https://mlsanity.com/paper/2507.23194/) (2025)
- [The Landscape and Challenges of HPC Research and LLMs](https://mlsanity.com/paper/2402.02018/) (2024)
- [Performance-Aligned LLMs for Generating Fast Code](https://mlsanity.com/paper/2404.18864/) (2024)
- [CodeRosetta: Pushing the Boundaries of Unsupervised Code Translation for Parallel Programming](https://mlsanity.com/paper/2410.20527/) (2024)
- [Towards Automated Kernel Generation in the Era of LLMs](https://mlsanity.com/paper/2601.15727/) (2026)
- [LASSI: An LLM-based Automated Self-Correcting Pipeline for Translating Parallel Scientific Codes](https://mlsanity.com/paper/2407.01638/) (2024)
- [HPC-Coder-V2: Studying Code LLMs Across Low-Resource Parallel Languages](https://mlsanity.com/paper/2412.15178/) (2024)
- [MPIrigen: MPI Code Generation through Domain-Specific Language Models](https://mlsanity.com/paper/2402.09126/) (2024)
- [OMPar: Automatic Parallelization with AI-Driven Source-to-Source Compilation](https://mlsanity.com/paper/2409.14771/) (2024)
- [Generative AI Uses and Risks for Knowledge Workers in a Science Organization](https://mlsanity.com/paper/2501.16577/) (2025)
- [MARCO: Multi-Agent Code Optimization with Real-Time Knowledge Integration for High-Performance Computing](https://mlsanity.com/paper/2505.03906/) (2025)
- [From Legacy Fortran to Portable Kokkos: An Autonomous Agentic AI Workflow](https://mlsanity.com/paper/2509.12443/) (2025)
- [More Than a Score: Probing the Impact of Prompt Specificity on LLM Code Generation](https://mlsanity.com/paper/2508.03678/) (2025)
- [Leveraging LLMs to Automate Energy-Aware Refactoring of Parallel Scientific Codes](https://mlsanity.com/paper/2505.02184/) (2025)
- [Beyond Code Pairs: Dialogue-Based Data Generation for LLM Code Translation](https://mlsanity.com/paper/2512.03086/) (2025)
- [PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization](https://mlsanity.com/paper/2607.19653/) (2026)
- [Leveraging AI for Productive and Trustworthy HPC Software: Challenges and Research Directions](https://mlsanity.com/paper/2505.08135/) (2025)
- [Performance Evaluation of General Purpose Large Language Models for Basic Linear Algebra Subprograms Code Generation](https://mlsanity.com/paper/2507.04697/) (2025)
- [Tutoring LLM into a Better CUDA Optimizer](https://mlsanity.com/paper/2510.16933/) (2025)
- [Improving HPC Code Generation Capability of LLMs via Online Reinforcement Learning with Real-Machine Benchmark Rewards](https://mlsanity.com/paper/2602.12049/) (2026)
- [Reinforced Fast Weights with Next-Sequence Prediction](https://mlsanity.com/paper/2602.16704/) (2026)
- [An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSc](https://mlsanity.com/paper/2603.15976/) (2026)
- [Not All Errors Are Equal: A Systematic Study of Error Propagation in Large Language Model Inference](https://mlsanity.com/paper/2606.02430/) (2026)
- [Exploring the Role of LLMs in HPC Programming: A Survey](https://mlsanity.com/paper/2608.26110/) (2026)
- [Modeling Code: Is Text All You Need?](https://mlsanity.com/paper/2507.11467/) (2025)
- [Accelerating Latency-Critical Applications with AI-Powered Semi-Automatic Fine-Grained Parallelization on SMT Processors](https://mlsanity.com/paper/2509.00883/) (2025)
- [OMPILOT: Harnessing Transformer Models for Auto Parallelization to Shared Memory Computing Paradigms](https://mlsanity.com/paper/2511.03866/) (2025)
- [HPCAgentTester: A Multi-Agent LLM Approach for Enhanced HPC Unit Test Generation](https://mlsanity.com/paper/2511.10860/) (2025)

## Abstract (arXiv)

Large language models are increasingly becoming a popular tool for software development. Their ability to model and generate source code has been demonstrated in a variety of contexts, including code completion, summarization, translation, and lookup. However, they often struggle to generate code for complex programs. In this paper, we study the capabilities of state-of-the-art language models to generate parallel code. In order to evaluate language models, we create a benchmark, ParEval, consisting of prompts that represent 420 different coding tasks related to scientific and parallel computing. We use ParEval to evaluate the effectiveness of several state-of-the-art open- and closed-source language models on these tasks. We introduce novel metrics for evaluating the performance of generated code, and use them to explore how well each large language model performs for 12 different computational problem types and six different parallel programming models.