mlsanityData to 7 Oct 2026

Twelve quick tips for designing AI-driven HPC workflows

arXiv 2606.07491 ↗First posted 5 Jun 2026cs.DC cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

High-performance computing (HPC) clusters remain the backbone of large-scale scientific computation, traditionally executing deterministic, linear pipelines optimised for predictable performance. However, the pervasive integration of artificial intelligence (AI) and foundation models into scientific research has introduced a fundamentally new computational paradigm. AI-driven workflows are characteristically iterative, data-driven, and probabilistic, introducing unique challenges regarding data gravity, heterogeneous resource management, and complex workflow orchestration. This guide provides twelve practical tips designed to help researchers design efficient, scalable, and reproducible AI-driven HPC workflows. By addressing critical system-level bottlenecks - such as containerisation for environment portability, strategic deployment of job arrays, explicit feedback loop mechanics, and I/O optimisation for small files - this article offers a framework for transitioning from rigid execution pipelines to adaptive, intelligent computational environments. While these architectural principles are broadly applicable across distributed environments, they are particularly tailored to the resource-intensive throughput demands of modern computational biology.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 10 of its 25 references that are in the map. First posted on arXiv 5 Jun 2026.

Builds on

10 of 25 references.

References · 10

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  3. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  4. 2021Membership Inference Attacks on Machine Learning: A Surveycited by 306
  5. 2019Parsl: Pervasive Parallel Programming in Pythoncited by 36
  6. 2024DataStates-LLM: Lazy Asynchronous Checkpointing for Large Language Modelscited by 16
  7. 2024Applying the FAIR Principles to computational workflowscited by 10
  8. 2022Enabling Dynamic and Intelligent Workflows for HPC, Data Analytics, and AI Convergencecited by 8
  9. 2024Workflows Community Summit 2024: Future Trends and Challenges in Scientific Workflowscited by 3
  10. 2021Ten Simple Rules for Success with HPC, i.e. Responsibly BASHing that Linux Clustercited by 1

Built on by

Not cited yet.