# ScaleFold: Reducing AlphaFold Initial Training Time to 10 Hours

arXiv: https://arxiv.org/abs/2404.11068 · HTML: https://mlsanity.com/paper/2404.11068/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 15 references that are in the map.
- First posted on arXiv 17 Apr 2024.

## Builds on (most cited first, 5 of 5)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [MLPerf HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systems](https://mlsanity.com/paper/2110.11466/) (2021)
- [FastFold: Reducing AlphaFold Training Time from 11 Days to 67 Hours](https://mlsanity.com/paper/2203.00854/) (2022)

## Built on by (most cited first, 8 of 8)

- [Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading](https://mlsanity.com/paper/2410.21316/) (2024)
- [A Survey on Memory-Efficient Transformer-Based Model Training in AI for Science](https://mlsanity.com/paper/2501.11847/) (2025)
- [MegaFold: Efficient Training of Next-Generation 3D Attention Protein Models on Cross-Platform GPUs](https://mlsanity.com/paper/2506.20686/) (2025)
- [Fold-CP: A Context Parallelism Framework for Biomolecular Modeling](https://mlsanity.com/paper/2603.14806/) (2026)
- [LightNobel: Improving Sequence Length Limitation in Protein Structure Prediction Model via Adaptive Activation Quantization](https://mlsanity.com/paper/2505.05893/) (2025)
- [Design-CP: Context Parallelism for Design of Protein Nanoparticles](https://mlsanity.com/paper/2607.05439/) (2026)
- [Structure-aware Reinforcement Learning for Protein Directed Evolution](https://mlsanity.com/paper/2609.39048/) (2026)
- [TurboPairFormer: Fast and Stable Protein Folding Model Training with an Optimized Triangle Attention Kernel](https://mlsanity.com/paper/2610.05854/) (2026)

## Abstract (arXiv)

AlphaFold2 has been hailed as a breakthrough in protein folding. It can rapidly predict protein structures with lab-grade accuracy. However, its implementation does not include the necessary training code. OpenFold is the first trainable public reimplementation of AlphaFold. AlphaFold training procedure is prohibitively time-consuming, and gets diminishing benefits from scaling to more compute resources. In this work, we conducted a comprehensive analysis on the AlphaFold training procedure based on Openfold, identified that inefficient communications and overhead-dominated computations were the key factors that prevented the AlphaFold training from effective scaling. We introduced ScaleFold, a systematic training method that incorporated optimizations specifically for these factors. ScaleFold successfully scaled the AlphaFold training to 2080 NVIDIA H100 GPUs with high resource utilization. In the MLPerf HPC v3.0 benchmark, ScaleFold finished the OpenFold benchmark in 7.51 minutes, shown over 6× speedup than the baseline. For training the AlphaFold model from scratch, ScaleFold completed the pretraining in 10 hours, a significant improvement over the seven days required by the original AlphaFold pretraining baseline.