Towards Decentralized and Sustainable Foundation Model Training with the Edge
Not on the map. Why?
Abstract (arXiv)
Foundation models are at the forefront of AI research, appealing for their ability to learn from vast datasets and cater to diverse tasks. Yet, their significant computational demands raise issues of environmental impact and the risk of centralized control in their development. We put forward a vision towards decentralized and sustainable foundation model training that leverages the collective compute of sparingly used connected edge AI devices. We present the rationale behind our vision, particularly in support of its sustainability benefit. We further outline a set of challenges that need to be addressed to turn this vision into reality.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 36 of its 103 references that are in the map. First posted on arXiv 2 Jul 2025.
Builds on
36 of 103 references.
References · 36
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2020Learning to summarize from human feedbackcited by 2,588
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
- 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
- 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
24 more
- 2020Adaptive Federated Optimizationcited by 1,046
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
- 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
- 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 231
- 2023Continual Pre-training of Language Modelscited by 198
- 2022The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrinkcited by 184
- 2020Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Modelscited by 169
- 2022Compute Trends Across Three Eras of Machine Learningcited by 163
- 2019Convergence of Edge Computing and Deep Learning: A Comprehensive Surveycited by 120
- 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
- 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
- 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 59
- 2021Distributed Deep Learning in Open Collaborationscited by 58
- 2023ZeRO++: Extremely Efficient Collective Communication for Giant Model Trainingcited by 52
- 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
- 2020Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Expertscited by 43
- 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
- 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 30
- 2023Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collectionscited by 16
- 2024Asteroid: Resource-Efficient Hybrid Pipeline Parallelism for Collaborative DNN Training on Heterogeneous Edge Devicescited by 13
- 2021IrEne: Interpretable Energy Prediction for Transformerscited by 11
- 2019BOINC: A Platform for Volunteer Computingcited by 6
- 2021Junkyard Computing: Repurposing Discarded Smartphones to Minimize Carboncited by 6
- 2022Rethinking Data-driven Networking with Foundation Models: Challenges and Opportunitiescited by 6
Built on by
Citing papers per year: 2025: 2.
2 papers.