mlsanityData to 14 Oct 2026

Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generation

arXiv 2306.15121 ↗First posted 27 Jun 2023cs.AI cs.ET cs.PLcited by 22

Not on the map. Why?

Abstract (arXiv)

We evaluate AI-assisted generative capabilities on fundamental numerical kernels in high-performance computing (HPC), including AXPY, GEMV, GEMM, SpMV, Jacobi Stencil, and CG. We test the generated kernel codes for a variety of language-supported programming models, including (1) C++ (e.g., OpenMP [including offload], OpenACC, Kokkos, SyCL, CUDA, and HIP), (2) Fortran (e.g., OpenMP [including offload] and OpenACC), (3) Python (e.g., numba, Numba, cuPy, and pyCUDA), and (4) Julia (e.g., Threads, CUDA.jl, AMDGPU.jl, and KernelAbstractions.jl). We use the GitHub Copilot capabilities powered by OpenAI Codex available in Visual Studio Code as of April 2023 to generate a vast amount of implementations given simple <kernel> + <programming model> + <optional hints> prompt variants. To quantify and compare the results, we propose a proficiency metric around the initial 10 suggestions given for each prompt. Results suggest that the OpenAI Codex outputs for C++ correlate with the adoption and maturity of programming models. For example, OpenMP and CUDA score really high, whereas HIP is still lacking. We found that prompts from either a targeted language such as Fortran or the more general-purpose Python can benefit from adding code keywords, while Julia prompts perform acceptably well for its mature programming models (e.g., Threads and CUDA.jl). We expect for these benchmarks to provide a point of reference for each programming model's community. Overall, understanding the convergence of large language models, AI, and HPC is crucial due to its rapidly evolving nature and how it is redefining human-computer interactions.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 11 of its 52 references that are in the map. First posted on arXiv 27 Jun 2023.

Builds on

11 of 52 references.

References · 11

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2019Generalizing from a Few Examples: A Survey on Few-Shot Learningcited by 463
  4. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 431
  5. 2022Automatic Generation of Programming Exercises and Code Explanations using Large Language Modelscited by 126
  6. 2022Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Languagecited by 82
  7. 2021Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programmingcited by 24
  8. 2017Effective Extensible Programming: Unleashing Julia on GPUscited by 16
  9. 2021PSB2: The Second Program Synthesis Benchmark Suitecited by 9
  10. 2023Evaluating performance and portability of high-level programming models: Julia, Python/Numba, and Kokkos on exascale nodescited by 2
  11. 2023PyExaFMM: an exercise in designing high-performance software with Python and Numbacited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 6; 2025: 8; 2026: 5.

22 papers.

Sort:

By area:

  1. 2024A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languagescited by 66
  2. 2024Can Large Language Models Write Parallel Code?cited by 36
  3. 2023Comparing Llama-2 and GPT-3 LLMs for HPC kernels generationcited by 30
  4. 2023HPC-GPT: Integrating Large Language Model for High-Performance Computingcited by 22
  5. 2024The Landscape and Challenges of HPC Research and LLMscited by 22
  6. 2023LLM4VV: Developing LLM-Driven Testsuite for Compiler Validationcited by 15
  7. 2026KernelFoundry: Hardware-aware evolutionary GPU kernel optimizationcited by 13
  8. 2025LLM & HPC:Benchmarking DeepSeek's Performance in High-Performance Computing Taskscited by 8
  9. 2026CuTeGen: An LLM-Based Agentic Framework for Generation and Optimization of High-Performance GPU Kernels using CuTecited by 6
  10. 2024MPIrigen: MPI Code Generation through Domain-Specific Language Modelscited by 5
  11. 2025Generative AI Uses and Risks for Knowledge Workers in a Science Organizationcited by 5
  12. 2025From Legacy Fortran to Portable Kokkos: An Autonomous Agentic AI Workflowcited by 5
10 more
  1. 2024Harnessing the Power of LLMs: Automating Unit Test Generation for High-Performance Computingcited by 3
  2. 2025Leveraging LLMs to Automate Energy-Aware Refactoring of Parallel Scientific Codescited by 3
  3. 2024SuperCode: Sustainability PER AI-driven CO-DEsigncited by 2
  4. 2025Leveraging AI for Productive and Trustworthy HPC Software: Challenges and Research Directionscited by 2
  5. 2025CelloAI: Leveraging Large Language Models for HPC Software Development in High Energy Physicscited by 2
  6. 2025VibeCodeHPC: An Agent-Based Iterative Prompting Auto-Tuner for HPC Code Generation Using LLMscited by 2
  7. 2025Performance Evaluation of General Purpose Large Language Models for Basic Linear Algebra Subprograms Code Generationcited by 1
  8. 2026ParaCodex: A Profiling-Guided Autonomous Coding Agent for Reliable Parallel Code Generation and Translationcited by 0
  9. 2026From Prompts to Performance: Evaluating LLMs for Task-based Parallel Code Generationcited by 0
  10. 2026AstraAI: LLMs, Retrieval, and AST-Guided Assistance for HPC Codebasescited by 0