mlsanityData to 7 Oct 2026

The Landscape and Challenges of HPC Research and LLMs

arXiv 2402.02018 ↗First posted 3 Feb 2024cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

Recently, language models (LMs), especially large language models (LLMs), have revolutionized the field of deep learning. Both encoder-decoder models and prompt-based techniques have shown immense potential for natural language processing and code-based tasks. Over the past several years, many research labs and institutions have invested heavily in high-performance computing, approaching or breaching exascale performance levels. In this paper, we posit that adapting and utilizing such language model-based techniques for tasks in high-performance computing (HPC) would be very beneficial. This study presents our reasoning behind the aforementioned position and highlights how existing ideas can be improved and adapted for HPC tasks.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 54 of its 89 references that are in the map. First posted on arXiv 3 Feb 2024.

Builds on

54 of 89 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
38 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  2. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  3. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  4. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  5. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  6. 2023StarCoder: may the source be with you!cited by 790
  7. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  8. 2020Heterogeneous Graph Transformercited by 581
  9. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  10. 2022Multimodal Learning with Transformers: A Surveycited by 399
  11. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  12. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  13. 2023Challenges and Applications of Large Language Modelscited by 251
  14. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 193
  15. 2022Automatic Generation of Programming Exercises and Code Explanations using Large Language Modelscited by 124
  16. 2021BERT: A Review of Applications in Natural Language Processing and Understandingcited by 109
  17. 2022Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Bookcited by 84
  18. 2023Large Language Models for Compiler Optimizationcited by 52
  19. 2023AI-assisted coding: Experiments with GPT-4cited by 43
  20. 2022Automatic Code Documentation Generation Using GPT-3cited by 37
  21. 2024Can Large Language Models Write Parallel Code?cited by 36
  22. 2023Comparing Llama-2 and GPT-3 LLMs for HPC kernels generationcited by 30
  23. 2023Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generationcited by 22
  24. 2023HPC-GPT: Integrating Large Language Model for High-Performance Computingcited by 22
  25. 2023LM4HPC: Towards Effective Language Model Application in High-Performance Computingcited by 18
  26. 2023Learning to Parallelize with OpenMP by Augmented Heterogeneous AST Representationcited by 14
  27. 2023Scope is all you need: Transforming LLMs for HPC Codecited by 12
  28. 2023MonoCoder: Domain-Specific Code Language Model for HPC Codes and Taskscited by 12
  29. 2021Enabling NAS with Automated Super-Network Generationcited by 11
  30. 2023HPC-Coder: Modeling Parallel Programs using Large Language Modelscited by 11
  31. 2023Data Race Detection Using Large Language Modelscited by 11
  32. 2022Learning to Parallelize in a Shared-Memory Environment with Transformerscited by 10
  33. 2023Advising OpenMP Parallelization via a Graph-Based Approach with Transformerscited by 10
  34. 2022Learning Intermediate Representations using Graph Neural Networks for NUMA and Prefetchers Optimizationcited by 9
  35. 2023Quantifying OpenMP: Statistical Insights into Usage and Adoptioncited by 9
  36. 2023Bridging the Gap Between Foundation Models and Heterogeneous Federated Learningcited by 9
  37. 2020ComPar: Optimized Multi-Compiler for Automatic OpenMP S2S Parallelizationcited by 8
  38. 2023Performance Optimization using Multimodal Modeling and Heterogeneous GNNcited by 7

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 11; 2026: 5.

22 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Models for Code Generationcited by 646
  2. 2025Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applicationscited by 26
  3. 2024Hybrid Heterogeneous Clusters Can Lower the Energy Consumption of LLM Inference Workloadscited by 17
  4. 2024Adaptive Layer Splitting for Wireless LLM Inference in Edge Computing: A Model-Based Reinforcement Learning Approachcited by 12
  5. 2025LLM & HPC:Benchmarking DeepSeek's Performance in High-Performance Computing Taskscited by 8
  6. 2024OMPar: Automatic Parallelization with AI-Driven Source-to-Source Compilationcited by 5
  7. 2024LLM as HPC Expert: Extending RAG Architecture for HPC Datacited by 5
  8. 2025MARCO: Multi-Agent Code Optimization with Real-Time Knowledge Integration for High-Performance Computingcited by 5
  9. 2024Can Large Language Model Predict Employee Attrition?cited by 3
  10. 2025Evaluating the Efficacy of LLM-Based Reasoning for Multiobjective HPC Job Schedulingcited by 3
  11. 2025Beyond Code Pairs: Dialogue-Based Data Generation for LLM Code Translationcited by 3
  12. 2025Leveraging AI for Productive and Trustworthy HPC Software: Challenges and Research Directionscited by 2
10 more
  1. 2025Counting Without Running: Evaluating LLMs' Reasoning About Code Complexitycited by 2
  2. 2025Scalable Runtime Architecture for Data-driven, Hybrid HPC and ML Workflow Applicationscited by 0
  3. 2025Evaluating the Safety and Skill Reasoning of Large Reasoning Models Under Compute Constraintscited by 0
  4. 2025OMPILOT: Harnessing Transformer Models for Auto Parallelization to Shared Memory Computing Paradigmscited by 0
  5. 2025Automated Dynamic AI Inference Scaling on HPC-Infrastructure: Integrating Kubernetes, Slurm and vLLMcited by 0
  6. 2026ParaCodex: A Profiling-Guided Autonomous Coding Agent for Reliable Parallel Code Generation and Translationcited by 0
  7. 2026AstraAI: LLMs, Retrieval, and AST-Guided Assistance for HPC Codebasescited by 0
  8. 2026Learning Reasoning World Models for Parallel Codecited by 0
  9. 2026HARGO: Heterogeneity-Aware Reward-Guided Optimization for RL Post-Training of LLMs on HPC Taskscited by 0
  10. 2026Backdoor Learning in Language Models and Vision-Language Modelscited by 0