mlsanityData to 7 Oct 2026

Can Large Language Models Write Parallel Code?

arXiv 2401.12554 ↗First posted 23 Jan 2024cs.DC cs.AIcited by 36

Not on the map. Why?

Abstract (arXiv)

Large language models are increasingly becoming a popular tool for software development. Their ability to model and generate source code has been demonstrated in a variety of contexts, including code completion, summarization, translation, and lookup. However, they often struggle to generate code for complex programs. In this paper, we study the capabilities of state-of-the-art language models to generate parallel code. In order to evaluate language models, we create a benchmark, ParEval, consisting of prompts that represent 420 different coding tasks related to scientific and parallel computing. We use ParEval to evaluate the effectiveness of several state-of-the-art open- and closed-source language models on these tasks. We introduce novel metrics for evaluating the performance of generated code, and use them to explore how well each large language model performs for 12 different computational problem types and six different parallel programming models.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 23 of its 54 references that are in the map. First posted on arXiv 23 Jan 2024.

Builds on

23 of 54 references.

References · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  11. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  12. 2022The Stack: 3 TB of permissively licensed source codecited by 354
11 more
  1. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 246
  2. 2023SantaCoder: don't reach for the stars!cited by 169
  3. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  4. 2020A Transformer-based Approach for Source Code Summarizationcited by 129
  5. 2023Comparing Llama-2 and GPT-3 LLMs for HPC kernels generationcited by 30
  6. 2023BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Modelscited by 24
  7. 2023Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generationcited by 22
  8. 2023LLM4VV: Developing LLM-Driven Testsuite for Compiler Validationcited by 15
  9. 2022Learning to Reduce False Positives in Analytic Bug Detectorscited by 12
  10. 2023Scope is all you need: Transforming LLMs for HPC Codecited by 12
  11. 2022Learning code summarization from a small and local datasetcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 15; 2026: 13.

36 papers.

Sort:

By area:

  1. 2024How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmarkcited by 39
  2. 2025LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarkscited by 34
  3. 2025Geak: Introducing Triton Kernel AI Agent & Evaluation Benchmarkscited by 31
  4. 2024The Landscape and Challenges of HPC Research and LLMscited by 22
  5. 2024Performance-Aligned LLMs for Generating Fast Codecited by 20
  6. 2024CodeRosetta: Pushing the Boundaries of Unsupervised Code Translation for Parallel Programmingcited by 19
  7. 2024LASSI: An LLM-based Automated Self-Correcting Pipeline for Translating Parallel Scientific Codescited by 10
  8. 2026Towards Automated Kernel Generation in the Era of LLMscited by 9
  9. 2024HPC-Coder-V2: Studying Code LLMs Across Low-Resource Parallel Languagescited by 8
  10. 2024MPIrigen: MPI Code Generation through Domain-Specific Language Modelscited by 5
  11. 2024OMPar: Automatic Parallelization with AI-Driven Source-to-Source Compilationcited by 5
  12. 2025Generative AI Uses and Risks for Knowledge Workers in a Science Organizationcited by 5
24 more
  1. 2025MARCO: Multi-Agent Code Optimization with Real-Time Knowledge Integration for High-Performance Computingcited by 5
  2. 2025More Than a Score: Probing the Impact of Prompt Specificity on LLM Code Generationcited by 4
  3. 2025From Legacy Fortran to Portable Kokkos: An Autonomous Agentic AI Workflowcited by 4
  4. 2025Leveraging LLMs to Automate Energy-Aware Refactoring of Parallel Scientific Codescited by 3
  5. 2025Beyond Code Pairs: Dialogue-Based Data Generation for LLM Code Translationcited by 3
  6. 2025Leveraging AI for Productive and Trustworthy HPC Software: Challenges and Research Directionscited by 2
  7. 2026PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimizationcited by 2
  8. 2025Performance Evaluation of General Purpose Large Language Models for Basic Linear Algebra Subprograms Code Generationcited by 1
  9. 2026Reinforced Fast Weights with Next-Sequence Predictioncited by 1
  10. 2026Not All Errors Are Equal: A Systematic Study of Error Propagation in Large Language Model Inferencecited by 1
  11. 2026Exploring the Role of LLMs in HPC Programming: A Surveycited by 1
  12. 2025Modeling Code: Is Text All You Need?cited by 0
  13. 2025Accelerating Latency-Critical Applications with AI-Powered Semi-Automatic Fine-Grained Parallelization on SMT Processorscited by 0
  14. 2025Tutoring LLM into a Better CUDA Optimizercited by 0
  15. 2025OMPILOT: Harnessing Transformer Models for Auto Parallelization to Shared Memory Computing Paradigmscited by 0
  16. 2025HPCAgentTester: A Multi-Agent LLM Approach for Enhanced HPC Unit Test Generationcited by 0
  17. 2026The New Compiler Stack: A Survey on the Synergy of LLMs and Compilerscited by 0
  18. 2026ParaCodex: A Profiling-Guided Autonomous Coding Agent for Reliable Parallel Code Generation and Translationcited by 0
  19. 2026Improving HPC Code Generation Capability of LLMs via Online Reinforcement Learning with Real-Machine Benchmark Rewardscited by 0
  20. 2026From Prompts to Performance: Evaluating LLMs for Task-based Parallel Code Generationcited by 0
  21. 2026ParEVO: Synthesizing Code for Irregular Data: High-Performance Parallelism through Agentic Evolutioncited by 0
  22. 2026An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSccited by 0
  23. 2026CodegenBench: Can LLMs Write Efficient Code Across Architectures?cited by 0
  24. 2026Cross-Model Cross-Language AI Coding Agent Performance: Accuracy and Speed of Parallel CLRS Algorithmscited by 0