# Distributed and Decentralised Training: Technical Governance Challenges in a Shifting AI Landscape

arXiv: https://arxiv.org/abs/2507.07765 · HTML: https://mlsanity.com/paper/2507.07765/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 44 references that are in the map.
- First posted on arXiv 10 Jul 2025.

## Builds on (most cited first, 19 of 19)

- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [DiLoCo: Distributed Low-Communication Training of Language Models](https://mlsanity.com/paper/2311.08105/) (2023)
- [A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility](https://mlsanity.com/paper/2504.07086/) (2025)
- [On the Societal Impact of Open Foundation Models](https://mlsanity.com/paper/2403.07918/) (2024)
- [Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development](https://mlsanity.com/paper/2501.16946/) (2025)
- [Computing Power and the Governance of Artificial Intelligence](https://mlsanity.com/paper/2402.08797/) (2024)
- [Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices](https://mlsanity.com/paper/2103.03239/) (2021)
- [The Future of Large Language Model Pre-training is Federated](https://mlsanity.com/paper/2405.10853/) (2024)
- [Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch](https://mlsanity.com/paper/2501.18512/) (2025)
- [Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo](https://mlsanity.com/paper/2503.09799/) (2025)
- [Asynchronous Local-SGD Training for Language Modeling](https://mlsanity.com/paper/2401.09135/) (2024)
- [The Shutdown Problem: An AI Engineering Puzzle for Decision Theorists](https://mlsanity.com/paper/2403.04471/) (2024)
- [INTELLECT-1 Technical Report](https://mlsanity.com/paper/2412.01152/) (2024)
- [TOPLOC: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference](https://mlsanity.com/paper/2501.16007/) (2025)
- [Oversight for Frontier AI through a Know-Your-Customer Scheme for Compute Providers](https://mlsanity.com/paper/2310.13625/) (2023)
- [Eager Updates For Overlapped Communication and Computation in DiLoCo](https://mlsanity.com/paper/2502.12996/) (2025)
- [Increased Compute Efficiency and the Diffusion of AI Capabilities](https://mlsanity.com/paper/2311.15377/) (2023)
- [AI Governance to Avoid Extinction: The Strategic Landscape and Actionable Research Questions](https://mlsanity.com/paper/2505.04592/) (2025)
- [Protocol Learning, Decentralized Frontier Risk and the No-Off Problem](https://mlsanity.com/paper/2412.07890/) (2024)

## Built on by (most cited first, 3 of 3)

- [Does Distributed Training Undermine Compute Governance?](https://mlsanity.com/paper/2605.29359/) (2026)
- [The Closing Window: How Governments Could Lose Their Ability to Restrain Advanced AI](https://mlsanity.com/paper/2608.05173/) (2026)
- [Beyond Training: A Feasibility Taxonomy for Inference-Time AI Governance](https://mlsanity.com/paper/2609.10105/) (2026)

## Abstract (arXiv)

Advances in low-communication training algorithms are enabling a shift from centralised model training to compute setups that are either distributed across multiple clusters or decentralised via community-driven contributions. This paper distinguishes these two scenarios - distributed and decentralised training - which are little understood and often conflated in policy discourse. We discuss how they could impact technical AI governance through an increased risk of compute structuring, capability proliferation, and the erosion of detectability and shutdownability. While these trends foreshadow a possible new paradigm that could challenge key assumptions of compute governance, we emphasise that certain policy levers, like export controls, remain relevant. We also acknowledge potential benefits of decentralised AI, including privacy-preserving training runs that could unlock access to more data, and mitigating harmful power concentration. Our goal is to support more precise policymaking around compute, capability proliferation, and decentralised AI development.