mlsanityData to 14 Oct 2026

No Need to Talk: Asynchronous Mixture of Language Models

arXiv 2410.03529 ↗First posted 4 Oct 2024cs.LG cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

We introduce SMALLTALK LM, an innovative method for training a mixture of language models in an almost asynchronous manner. Each model of the mixture specializes in distinct parts of the data distribution, without the need for high-bandwidth communication between the nodes training each model. At inference, a lightweight router directs a given sequence to a single expert, according to a short prefix. This inference scheme naturally uses a fraction of the parameters from the overall mixture model. Unlike prior works on asynchronous LLM training, our routing method does not rely on full corpus clustering or access to metadata, making it more suitable for real-world applications. Our experiments on language modeling demonstrate that SMALLTALK LM achieves significantly lower perplexity than dense model baselines for the same total training FLOPs and an almost identical inference cost. Finally, in our downstream evaluations we outperform the dense baseline on 75% of the tasks.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 43 of its 58 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

43 of 58 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
31 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  3. 2024Mixtral of Expertscited by 1,823
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  5. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  6. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  7. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  8. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  9. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  10. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  11. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
  12. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  13. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
  14. 2017Sparse Communication for Distributed Gradient Descentcited by 249
  15. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  16. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  17. 2017Asynchronous Decentralized Parallel Stochastic Gradient Descentcited by 159
  18. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 121
  19. 2014Deep learning with Elastic Averaging SGDcited by 90
  20. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 87
  21. 2015Distributed Gaussian Processescited by 84
  22. 2018Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGDcited by 77
  23. 2017Hard Mixtures of Experts for Large Scale Weakly Supervised Visioncited by 67
  24. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 60
  25. 2021Distributed Deep Learning in Open Collaborationscited by 58
  26. 20158-Bit Approximations for Parallelism in Deep Learningcited by 56
  27. 2015Staleness-aware Async-SGD for Distributed Deep Learningcited by 43
  28. 2021Scaling Federated Learning for Fine-tuning of Large Language Modelscited by 36
  29. 2015Generative Image Modeling Using Spatial LSTMscited by 35
  30. 2024Asynchronous Local-SGD Training for Language Modelingcited by 26
  31. 2022Scaling Language Model Size in Cross-Device Federated Learningcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026Latent Space Communication via K-V Cache Alignmentcited by 17
  2. 2025BTS: Harmonizing Specialized Experts into a Generalist LLMcited by 6
  3. 2026Optimal Splitting of Language Models from Mixtures to Specialized Domainscited by 1
  4. 2026Decentralized Autoregressive Generationcited by 0
  5. 2026Scaling Zero-Order Pretraining through Model Shardingcited by 0