mlsanityData to 7 Oct 2026

HPC-Coder-V2: Studying Code LLMs Across Low-Resource Parallel Languages

arXiv 2412.15178 ↗First posted 19 Dec 2024cs.DC cs.LG cs.SEcited by 8

Not on the map. Why?

Abstract (arXiv)

Large Language Model (LLM) based coding tools have been tremendously successful as software development assistants, yet they are often designed for general purpose programming tasks and perform poorly for more specialized domains such as high performance computing. Creating specialized models and tools for these domains is crucial towards gaining the benefits of LLMs in areas such as HPC. While previous work has explored HPC-specific models, LLMs still struggle to generate parallel code and it is not at all clear what hurdles are still holding back these LLMs and what must be done to overcome them. In this work, we conduct an in-depth study along the many axes of fine-tuning a specialized HPC LLM in order to better understand the challenges. Based on our findings we fine-tune and evaluate a specialized HPC LLM that is shown to be the best performing open-source code LLM for parallel code generation to date.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 28 of its 40 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

28 of 40 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  9. 2017Decoupled Weight Decay Regularizationcited by 1,722
  10. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  11. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  12. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
16 more
  1. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  2. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  3. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  4. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 246
  5. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  6. 2023Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMscited by 45
  7. 2024Can Large Language Models Write Parallel Code?cited by 36
  8. 2024Performance-Aligned LLMs for Generating Fast Codecited by 20
  9. 2024OMPGPT: A Generative Pre-trained Transformer Model for OpenMPcited by 16
  10. 2023LLM4VV: Developing LLM-Driven Testsuite for Compiler Validationcited by 15
  11. 2023Scope is all you need: Transforming LLMs for HPC Codecited by 12
  12. 2023HPC-Coder: Modeling Parallel Programs using Large Language Modelscited by 11
  13. 2023Data Race Detection Using Large Language Modelscited by 11
  14. 2024LASSI: An LLM-based Automated Self-Correcting Pipeline for Translating Parallel Scientific Codescited by 10
  15. 2023Exploiting Sparsity in Pruned Neural Networks to Optimize Large Model Trainingcited by 5
  16. 2024MPIrigen: MPI Code Generation through Domain-Specific Language Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 3.

8 papers.

  1. 2025Leveraging LLMs to Automate Energy-Aware Refactoring of Parallel Scientific Codescited by 3
  2. 2026OptiML: An End-to-End Framework for Program Synthesis and CUDA Kernel Optimizationcited by 3
  3. 2025Systematic Evaluation of Optimization Techniques for Long-Context Language Modelscited by 1
  4. 2025Integrating Performance Tools in Model Reasoning for GPU Kernel Optimizationcited by 1
  5. 2025Lessons Learned: A Multi-Agent Framework for Code LLMs to Learn and Improvecited by 0
  6. 2025OMPILOT: Harnessing Transformer Models for Auto Parallelization to Shared Memory Computing Paradigmscited by 0
  7. 2026An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSccited by 0
  8. 2026HARGO: Heterogeneity-Aware Reward-Guided Optimization for RL Post-Training of LLMs on HPC Taskscited by 0