mlsanityData to 14 Oct 2026

Training Transformers Together

arXiv 2207.03481 ↗First posted 7 Jul 2022cs.LG cs.DCcited by 5

Not on the map. Why?

Abstract (arXiv)

The infrastructure necessary for training state-of-the-art models is becoming overly expensive, which makes training such models affordable only to large corporations and institutions. Recent work proposes several methods for training such models collaboratively, i.e., by pooling together hardware from many independent parties and training a shared model over the Internet. In this demonstration, we collaboratively trained a text-to-image transformer similar to OpenAI DALL-E. We invited the viewers to join the ongoing training run, showing them instructions on how to contribute using the available hardware. We explained how to address the engineering challenges associated with such a training run (slow communication, limited memory, uneven performance between devices, and security concerns) and discussed how the viewers can set up collaborative training runs themselves. Finally, we show that the resulting model generates images of reasonable quality on a number of prompts.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 36 of its 51 references that are in the map. First posted on arXiv 7 Jul 2022.

Builds on

36 of 51 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
24 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  7. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  8. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  9. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
  10. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  11. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  12. 2016Using the Output Embedding to Improve Language Modelscited by 272
  13. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
  14. 2018Scaling Neural Machine Translationcited by 204
  15. 2020Learning from History for Byzantine Robust Optimizationcited by 137
  16. 2018Training Tips for the Transformer Modelcited by 85
  17. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 75
  18. 2021Distributed Deep Learning in Open Collaborationscited by 58
  19. 2021Go Wider Instead of Deepercited by 57
  20. 20158-Bit Approximations for Parallelism in Deep Learningcited by 56
  21. 2020Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Expertscited by 43
  22. 2019A Partially Reversible U-Net for Memory-Efficient Volumetric Image Segmentationcited by 16
  23. 2021Secure Distributed Training at Scalecited by 12
  24. 2021Distributed Deep Learning Using Volunteer Computing-Like Paradigmcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 0; 2025: 2; 2026: 1.

5 papers.

  1. 2023How Can We Train Deep Learning Models Across Clouds and Continents? An Experimental Studycited by 4
  2. 2025TAH-QUANT: Effective Activation Quantization in Pipeline Parallelism over Slow Networkcited by 4
  3. 2023Towards Open Federated Learning Platforms: Survey and Vision from Technical and Legal Perspectivescited by 3
  4. 2025Incentivizing Permissionless Distributed Learning of LLMscited by 3
  5. 2026Decentralised AI Training and Inference with BlockTraincited by 0