# Incentivizing Permissionless Distributed Learning of LLMs

arXiv: https://arxiv.org/abs/2505.21684 · HTML: https://mlsanity.com/paper/2505.21684/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 20 references that are in the map.
- First posted on arXiv 27 May 2025.

## Builds on (most cited first, 16 of 16)

- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Robust Aggregation for Federated Learning](https://mlsanity.com/paper/1912.13445/) (2019)
- [Error Feedback Fixes SignSGD and other Gradient Compression Schemes](https://mlsanity.com/paper/1901.09847/) (2019)
- [DiLoCo: Distributed Low-Communication Training of Language Models](https://mlsanity.com/paper/2311.08105/) (2023)
- [Generalized Byzantine-tolerant SGD](https://mlsanity.com/paper/1802.10116/) (2018)
- [Proof-of-Learning: Definitions and Practice](https://mlsanity.com/paper/2103.05633/) (2021)
- [Understanding Top-k Sparsification in Distributed Deep Learning](https://mlsanity.com/paper/1911.08772/) (2019)
- [Error Compensated Distributed SGD Can Be Accelerated](https://mlsanity.com/paper/2010.00091/) (2020)
- [The Future of Large Language Model Pre-training is Federated](https://mlsanity.com/paper/2405.10853/) (2024)
- [INTELLECT-1 Technical Report](https://mlsanity.com/paper/2412.01152/) (2024)
- [OpenSkill: A faster asymmetric multi-team, multiplayer rating system](https://mlsanity.com/paper/2401.05451/) (2024)
- [Verde: Verification via Refereed Delegation for Machine Learning Programs](https://mlsanity.com/paper/2502.19405/) (2025)
- [Training Transformers Together](https://mlsanity.com/paper/2207.03481/) (2022)
- [Byzantine Robustness and Partial Participation Can Be Achieved at Once: Just Clip Gradient Differences](https://mlsanity.com/paper/2311.14127/) (2023)

## Built on by (most cited first, 3 of 3)

- [Overcoming the Communication-Performance Tradeoff in LLM Pretraining](https://mlsanity.com/paper/2508.15706/) (2025)
- [Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet](https://mlsanity.com/paper/2603.08163/) (2026)
- [MAGNET: Autonomous Expert Model Generation via Decentralized Autoresearch and BitNet Training](https://mlsanity.com/paper/2603.25813/) (2026)

## Abstract (arXiv)

We describe an incentive system for distributed deep learning of foundational models where peers are rewarded for contributions. The incentive system, textitGauntlet, has been deployed on the bittensor blockchain and used to train a 1.2B LLM with completely permissionless contributions of pseudo-gradients: no control over the users that can register or their hardware. textitGauntlet can be applied to any synchronous distributed training scheme that relies on aggregating updates or pseudo-gradients. We rely on a two-stage mechanism for fast filtering of peer uptime, reliability, and synchronization, combined with the core component that estimates the loss before and after individual pseudo-gradient contributions. We utilized an OpenSkill rating system to track competitiveness of pseudo-gradient scores across time. Finally, we introduce a novel mechanism to ensure peers on the network perform unique computations. Our live 1.2B run, which has paid out real-valued tokens to participants based on the value of their contributions, yielded a competitive (on a per-iteration basis) 1.2B model that demonstrates the utility of our incentive system.