# Fold-CP: A Context Parallelism Framework for Biomolecular Modeling

arXiv: https://arxiv.org/abs/2603.14806 · HTML: https://mlsanity.com/paper/2603.14806/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 31 references that are in the map.
- First posted on arXiv 16 Mar 2026.

## Builds on (most cited first, 5 of 5)

- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Ring Attention with Blockwise Transformers for Near-Infinite Context](https://mlsanity.com/paper/2310.01889/) (2023)
- [LASP-2: Rethinking Sequence Parallelism for Linear Attention and Its Hybrid](https://mlsanity.com/paper/2502.07563/) (2025)
- [ScaleFold: Reducing AlphaFold Initial Training Time to 10 Hours](https://mlsanity.com/paper/2404.11068/) (2024)
- [APACE: AlphaFold2 and advanced computing as a service for accelerated discovery in biophysics](https://mlsanity.com/paper/2308.07954/) (2023)

## Built on by (most cited first, 1 of 1)

- [Design-CP: Context Parallelism for Design of Protein Nanoparticles](https://mlsanity.com/paper/2607.05439/) (2026)

## Abstract (arXiv)

Understanding cellular machinery requires atomic-scale reconstruction of large biomolecular assemblies. However, predicting the structures of these systems has been constrained by hardware memory requirements of models like AlphaFold 3, imposing a practical ceiling of a few thousand residues that can be processed on a single GPU. Here we present NVIDIA BioNeMo Fold-CP, a context parallelism framework that overcomes this barrier by distributing the inference and training pipelines of co-folding models across multiple GPUs. We use the Boltz models as open source reference architectures and implement custom multidimensional primitives that efficiently parallelize both the dense triangular updates and the irregular, data-dependent pattern of window-batched local attention. Our approach achieves efficient memory scaling; for an N-token input distributed across P GPUs, per-device memory scales as O(N2/P), enabling the structure prediction of assemblies exceeding 30,000 residues on 64 NVIDIA B300 GPUs. We demonstrate the scientific utility of this approach through successful developer use cases: Fold-CP enabled the scoring of over 90% of Comprehensive Resource of Mammalian protein complexes (CORUM) database, as well as folding of disease-relevant PI4KA lipid kinase complex bound to an intrinsically disordered region without cropping. By providing a scalable pathway for modeling massive systems with full global context, Fold-CP represents a significant step toward the realization of a virtual cell.