# Comparing Llama-2 and GPT-3 LLMs for HPC kernels generation

arXiv: https://arxiv.org/abs/2309.07103 · HTML: https://mlsanity.com/paper/2309.07103/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 24 references that are in the map.
- First posted on arXiv 12 Sep 2023.

## Builds on (most cited first, 7 of 7)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generation](https://mlsanity.com/paper/2306.15121/) (2023)
- [LM4HPC: Towards Effective Language Model Application in High-Performance Computing](https://mlsanity.com/paper/2306.14979/) (2023)
- [Effective Extensible Programming: Unleashing Julia on GPUs](https://mlsanity.com/paper/1712.03112/) (2017)
- [HPC-Coder: Modeling Parallel Programs using Large Language Models](https://mlsanity.com/paper/2306.17281/) (2023)
- [Evaluating performance and portability of high-level programming models: Julia, Python/Numba, and Kokkos on exascale nodes](https://mlsanity.com/paper/2303.06195/) (2023)

## Built on by (most cited first, 30 of 30)

- [KernelBench: Can LLMs Write Efficient GPU Kernels?](https://mlsanity.com/paper/2502.10517/) (2025)
- [Can Large Language Models Write Parallel Code?](https://mlsanity.com/paper/2401.12554/) (2024)
- [The Landscape and Challenges of HPC Research and LLMs](https://mlsanity.com/paper/2402.02018/) (2024)
- [Performance-Aligned LLMs for Generating Fast Code](https://mlsanity.com/paper/2404.18864/) (2024)
- [OMPGPT: A Generative Pre-trained Transformer Model for OpenMP](https://mlsanity.com/paper/2401.16445/) (2024)
- [LLM4VV: Developing LLM-Driven Testsuite for Compiler Validation](https://mlsanity.com/paper/2310.04963/) (2023)
- [KernelFoundry: Hardware-aware evolutionary GPU kernel optimization](https://mlsanity.com/paper/2603.12440/) (2026)
- [Temporal Knowledge Graph Question Answering: A Survey](https://mlsanity.com/paper/2406.14191/) (2024)
- [LLM & HPC:Benchmarking DeepSeek's Performance in High-Performance Computing Tasks](https://mlsanity.com/paper/2504.03665/) (2025)
- [LLM4GRN: Discovering Causal Gene Regulatory Networks with LLMs -- Evaluation through Synthetic Data Generation](https://mlsanity.com/paper/2410.15828/) (2024)
- [CuTeGen: An LLM-Based Agentic Framework for Generation and Optimization of High-Performance GPU Kernels using CuTe](https://mlsanity.com/paper/2604.01489/) (2026)
- [MPIrigen: MPI Code Generation through Domain-Specific Language Models](https://mlsanity.com/paper/2402.09126/) (2024)
- [OMPar: Automatic Parallelization with AI-Driven Source-to-Source Compilation](https://mlsanity.com/paper/2409.14771/) (2024)
- [Leveraging AI for Productive and Trustworthy HPC Software: Challenges and Research Directions](https://mlsanity.com/paper/2505.08135/) (2025)
- [VibeCodeHPC: An Agent-Based Iterative Prompting Auto-Tuner for HPC Code Generation Using LLMs](https://mlsanity.com/paper/2510.00031/) (2025)
- [Should AI Optimize Your Code? A Comparative Study of Classical Optimizing Compilers Versus Current Large Language Models](https://mlsanity.com/paper/2406.12146/) (2024)
- [Event Segmentation Applications in Large Language Model Enabled Automated Recall Assessments](https://mlsanity.com/paper/2502.13349/) (2025)
- [AI Assistants to Enhance and Exploit the PETSc Knowledge Base](https://mlsanity.com/paper/2506.20608/) (2025)
- [Performance Evaluation of General Purpose Large Language Models for Basic Linear Algebra Subprograms Code Generation](https://mlsanity.com/paper/2507.04697/) (2025)
- [An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSc](https://mlsanity.com/paper/2603.15976/) (2026)
- [Exploring the Role of LLMs in HPC Programming: A Survey](https://mlsanity.com/paper/2608.26110/) (2026)
- [Automated Creation of Source Code Variants of a Cryptographic Hash Function Implementation Using Generative Pre-Trained Transformer Models](https://mlsanity.com/paper/2404.15681/) (2024)
- [EPIC: Generative AI Platform for Accelerating HPC Operational Data Analytics](https://mlsanity.com/paper/2509.16212/) (2025)
- [ParaCodex: A Profiling-Guided Autonomous Coding Agent for Reliable Parallel Code Generation and Translation](https://mlsanity.com/paper/2601.04327/) (2026)
- [From Prompts to Performance: Evaluating LLMs for Task-based Parallel Code Generation](https://mlsanity.com/paper/2602.22240/) (2026)
- [AstraAI: LLMs, Retrieval, and AST-Guided Assistance for HPC Codebases](https://mlsanity.com/paper/2603.27423/) (2026)
- [Learning Reasoning World Models for Parallel Code](https://mlsanity.com/paper/2604.20926/) (2026)
- [CodegenBench: Can LLMs Write Efficient Code Across Architectures?](https://mlsanity.com/paper/2606.04023/) (2026)
- [Cross-Model Cross-Language AI Coding Agent Performance: Accuracy and Speed of Parallel CLRS Algorithms](https://mlsanity.com/paper/2607.26083/) (2026)
- [Effect of Abstractions and Prompting Strategies on LLM-Guided High-Performance Optimizations](https://mlsanity.com/paper/2608.08085/) (2026)

## Abstract (arXiv)

We evaluate the use of the open-source Llama-2 model for generating well-known, high-performance computing kernels (e.g., AXPY, GEMV, GEMM) on different parallel programming models and languages (e.g., C++: OpenMP, OpenMP Offload, OpenACC, CUDA, HIP; Fortran: OpenMP, OpenMP Offload, OpenACC; Python: numpy, Numba, pyCUDA, cuPy; and Julia: Threads, CUDA.jl, AMDGPU.jl). We built upon our previous work that is based on the OpenAI Codex, which is a descendant of GPT-3, to generate similar kernels with simple prompts via GitHub Copilot. Our goal is to compare the accuracy of Llama-2 and our original GPT-3 baseline by using a similar metric. Llama-2 has a simplified model that shows competitive or even superior accuracy. We also report on the differences between these foundational large language models as generative AI continues to redefine human-computer interactions. Overall, Copilot generates codes that are more reliable but less optimized, whereas codes generated by Llama-2 are less reliable but more optimized when correct.