# Exploring the Role of LLMs in HPC Programming: A Survey

arXiv: https://arxiv.org/abs/2608.26110 · HTML: https://mlsanity.com/paper/2608.26110/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 45 references that are in the map.
- First posted on arXiv 24 May 2026.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [Can Large Language Models Write Parallel Code?](https://mlsanity.com/paper/2401.12554/) (2024)
- [Comparing Llama-2 and GPT-3 LLMs for HPC kernels generation](https://mlsanity.com/paper/2309.07103/) (2023)
- [HPC-GPT: Integrating Large Language Model for High-Performance Computing](https://mlsanity.com/paper/2311.12833/) (2023)
- [Leveraging Large Language Models for Code Translation and Software Development in Scientific Computing](https://mlsanity.com/paper/2410.24119/) (2024)
- [ParEval-Repo: A Benchmark Suite for Evaluating LLMs with Repository-level HPC Translation Tasks](https://mlsanity.com/paper/2506.20938/) (2025)
- [UniPar: A Unified LLM-Based Framework for Parallel and Accelerated Code Translation in HPC](https://mlsanity.com/paper/2509.12136/) (2025)
- [Leveraging LLMs to Automate Energy-Aware Refactoring of Parallel Scientific Codes](https://mlsanity.com/paper/2505.02184/) (2025)
- [LLM-HPC++: Evaluating LLM-Generated Modern C++ and MPI+OpenMP Codes for Scalable Mandelbrot Set Computation](https://mlsanity.com/paper/2512.17023/) (2025)
- [P4OMP: Retrieval-Augmented Prompting for OpenMP Parallelism in Serial Code](https://mlsanity.com/paper/2506.22703/) (2025)
- [Adding New Capability in Existing Scientific Application with LLM Assistance](https://mlsanity.com/paper/2511.00087/) (2025)

## Built on by (most cited first, 1 of 1)

- [HARGO: Heterogeneity-Aware Reward-Guided Optimization for RL Post-Training of LLMs on HPC Tasks](https://mlsanity.com/paper/2607.28301/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are emerging as promising assistants in High-Performance Computing (HPC), where programming remains complex and expertise-intensive. This survey systematically reviews their application across five categories: code generation, parallelization and optimization, frameworks and architectures, evaluation and benchmarking, and broader challenges. The analysis highlights both opportunities and limitations: while general-purpose LLMs perform reasonably well on serial and OpenMP-like tasks, they fall short in distributed paradigms such as MPI, where correctness and scalability are critical. Domain-specialized models (e.g., HPC-Coder, HPC-GPT, chatHPC) achieve higher accuracy through fine-tuning, curated datasets, and retrieval-augmented generation (RAG), yet their scope remains narrow and their evaluations largely limited to benchmarks or micro-kernels. The broader picture is one of dual potential and fragility: LLMs can lower barriers to entry, accelerate prototyping, and support code modernization, but they remain brittle under production-level requirements where correctness, performance portability, and scaling cannot be compromised. We conclude that LLMs are unlikely to replace HPC experts in the near term but are positioned to become powerful collaborators in the software development pipeline. Their effective deployment will require richer datasets, integration with performance analysis and schedulers, rigorous evaluation frameworks, and governance structures that ensure transparency and trust. The convergence of AI and HPC should therefore be understood as a long-term, co-evolutionary process, where each advance uncovers new challenges and opportunities for reshaping scientific software development.