# INTELLECT-1 Technical Report

arXiv: https://arxiv.org/abs/2412.01152 · HTML: https://mlsanity.com/paper/2412.01152/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2024.

## Built on by (most cited first, 21 of 21)

- [Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch](https://mlsanity.com/paper/2501.18512/) (2025)
- [MuLoCo: Muon is a practical inner optimizer for DiLoCo](https://mlsanity.com/paper/2505.23725/) (2025)
- [Overcoming the Communication-Performance Tradeoff in LLM Pretraining](https://mlsanity.com/paper/2508.15706/) (2025)
- [Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Acceleration](https://mlsanity.com/paper/2509.10439/) (2025)
- [Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet](https://mlsanity.com/paper/2603.08163/) (2026)
- [NoLoCo: No-all-reduce Low Communication Training Method for Large Models](https://mlsanity.com/paper/2506.10911/) (2025)
- [HDEE: Heterogeneous Domain Expert Ensemble](https://mlsanity.com/paper/2502.19385/) (2025)
- [Incentivizing Permissionless Distributed Learning of LLMs](https://mlsanity.com/paper/2505.21684/) (2025)
- [Distributed and Decentralised Training: Technical Governance Challenges in a Shifting AI Landscape](https://mlsanity.com/paper/2507.07765/) (2025)
- [Few-Step Diffusion Language Models via Trajectory Self-Distillation](https://mlsanity.com/paper/2602.12262/) (2026)
- [DICE: Data Influence Cascade in Decentralized Learning](https://mlsanity.com/paper/2507.06931/) (2025)
- [Compute Requirements for Algorithmic Innovation in Frontier AI Models](https://mlsanity.com/paper/2507.10618/) (2025)
- [HybridEP: Scaling Expert Parallelism to Cross-Datacenter Scenario via Hybrid Expert/Data Transmission](https://mlsanity.com/paper/2510.19470/) (2025)
- [Can Model Merging Improve Aggregation in DiLoCo?](https://mlsanity.com/paper/2607.03011/) (2026)
- [Hesitation-Aware On-Policy Distillation for Diffusion Language Models](https://mlsanity.com/paper/2609.33301/) (2026)
- [Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm](https://mlsanity.com/paper/2602.11543/) (2026)
- [Local MixVR: Breaking the Communication-Sample Dependence in Distributed Learning](https://mlsanity.com/paper/2606.01128/) (2026)
- [Decentralised AI Training and Inference with BlockTrain](https://mlsanity.com/paper/2606.24722/) (2026)
- [ShardMeter: Sharded and Geo-Distributed Training Without the Guesswork](https://mlsanity.com/paper/2608.23840/) (2026)
- [Distributed Training using an Intelligent Network](https://mlsanity.com/paper/2608.26453/) (2026)
- [Greedy Local Learning for Language Model Pretraining: Gaps and Objective Design](https://mlsanity.com/paper/2610.04867/) (2026)

## Abstract (arXiv)

In this report, we introduce INTELLECT-1, the first 10 billion parameter language model collaboratively trained across the globe, demonstrating that large-scale model training is no longer confined to large corporations but can be achieved through a distributed, community-driven approach. INTELLECT-1 was trained on 1 trillion tokens using up to 14 concurrent nodes distributed across 3 continents, with contributions from 30 independent compute providers dynamically joining and leaving the training process, while maintaining 83-96% compute utilization and 36.2-41.4% model FLOPS utilization. We leverage PRIME, our scalable distributed training framework designed for fault-tolerant, high-performance training on unreliable, globally distributed nodes. Key innovations in PRIME include the ElasticDeviceMesh, which manages dynamic global process groups for fault-tolerant communication across the internet and local process groups for communication within a node, live checkpoint recovery kernels, and a hybrid DiLoCo-FSDP2 implementation. Using PRIME with DiLoCo and our custom int8 all-reduce, we achieve a 400x reduction in communication bandwidth compared to traditional data-parallel training settings while delivering comparable performance. These results demonstrate the feasibility and promise of training frontier foundation models in a decentralized network of global GPU resources.