mlsanityData to 14 Oct 2026

Scaling Expert Language Models with Unsupervised Domain Discovery

arXiv 2303.14177 ↗First posted 24 Mar 2023cs.CL cs.AIcited by 60

Not on the map. Why?

Abstract (arXiv)

Large language models are typically trained densely: all parameters are updated with respect to all inputs. This requires synchronization of billions of parameters across thousands of GPUs. We introduce a simple but effective method to asynchronously train large, sparse language models on arbitrary text corpora. Our method clusters a corpus into sets of related documents, trains a separate expert language model on each cluster, and combines them in a sparse ensemble for inference. This approach generalizes embarrassingly parallel training by automatically discovering the domains for each expert, and eliminates nearly all the communication overhead of existing sparse language models. Our technique outperforms dense baselines on multiple corpora and few-shot tasks, and our analysis shows that specializing experts to meaningful clusters is key to these gains. Performance also improves with the number of experts and size of training data, suggesting this is a highly efficient and accessible approach to training large language models.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Builds on 47 of its 62 references that are in the map. First posted on arXiv 24 Mar 2023.

Builds on

47 of 62 references.

References · 47

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  9. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  12. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
35 more
  1. 2022Galactica: A Large Language Model for Sciencecited by 744
  2. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  3. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  4. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 417
  5. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  6. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  7. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  8. 2018An Empirical Model of Large-Batch Trainingcited by 281
  9. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  10. 2021Hash Layers For Large Sparse Modelscited by 237
  11. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 213
  12. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  13. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
  14. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  15. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 189
  16. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  17. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  18. 2022A Review of Sparse Expert Models in Deep Learningcited by 148
  19. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  20. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 132
  21. 2021The Efficiency Misnomercited by 98
  22. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  23. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  24. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 87
  25. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
  26. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  27. 2017Hard Mixtures of Experts for Large Scale Weakly Supervised Visioncited by 67
  28. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 59
  29. 2017Hash Embeddings for Efficient Word Representationscited by 55
  30. 2021Efficient Hierarchical Domain Adaptation for Pretrained Language Modelscited by 32
  31. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 30
  32. 2021Characterizing English Variation across Social Media Communities with BERTcited by 20
  33. 2022lo-fi: distributed fine-tuning without communicationcited by 19
  34. 2021BAGUA: Scaling up Distributed Learning with System Relaxationscited by 13
  35. 2014Balanced k-Means and Min-Cut Clusteringcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 29; 2025: 11; 2026: 11.

60 papers.

Sort:

By area:

  1. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  2. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 184
  3. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 157
  4. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 121
  5. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 109
  6. 2023Efficient Online Data Mixing For Language Model Pre-Trainingcited by 77
  7. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 76
  8. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  9. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  10. 2023Guiding Language Model Reasoning with Planning Tokenscited by 58
  11. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  12. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
38 more
  1. 2024Breaking the Curse of Multilinguality with Cross-lingual Expert Language Modelscited by 38
  2. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 28
  3. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  4. 2023CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Modelscited by 26
  5. 2024AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filterscited by 22
  6. 2024MoDE: CLIP Data Experts via Clusteringcited by 22
  7. 2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21
  8. 2023Sabi'a: Portuguese Large Language Modelscited by 19
  9. 2024Contextual Document Embeddingscited by 19
  10. 2024SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Expertscited by 18
  11. 2023Token-Level Adaptation of LoRA Adapters for Downstream Task Generalizationcited by 17
  12. 2024AcTracer: Active Testing of Large Language Model via Multi-Stage Samplingcited by 17
  13. 2024PLeaS -- Merging Models with Permutations and Least Squarescited by 16
  14. 2024MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routingcited by 16
  15. 2025FlexOlmo: Open Language Models for Flexible Data Usecited by 15
  16. 2024LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Modelscited by 14
  17. 2025Pretraining with hierarchical memories: separating long-tail and common knowledgecited by 14
  18. 2025MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEscited by 13
  19. 2024Scalable Data Ablation Approximations for Language Models through Modular Training and Mergingcited by 11
  20. 2024DiPaCo: Distributed Path Compositioncited by 10
  21. 2024SynTQA: Synergistic Table-based Question Answering via Mixture of Text-to-SQL and E2E TQAcited by 9
  22. 2025Local Mixtures of Experts: Essentially Free Test-Time Training via Model Mergingcited by 9
  23. 2023Hazards from Increasingly Accessible Fine-Tuning of Downloadable Foundation Modelscited by 7
  24. 2024Nexus: Specialization meets Adaptability for Efficiently Training Mixture of Expertscited by 7
  25. 2024Copyright-Protected Language Generation via Adaptive Model Fusioncited by 7
  26. 2025BTS: Harmonizing Specialized Experts into a Generalist LLMcited by 6
  27. 2024M2QA: Multi-domain Multilingual Question Answeringcited by 5
  28. 2024No Need to Talk: Asynchronous Mixture of Language Modelscited by 5
  29. 2024What the Weight?! A Unified Framework for Zero-Shot Knowledge Compositioncited by 4
  30. 2024Tiny Models are the Computational Saver for Large Modelscited by 4
  31. 2024Automatic register identification for the open web using multilingual deep learningcited by 4
  32. 2024Strong Copyright Protection for Language Models via Adaptive Model Fusioncited by 4
  33. 2025Innovator: Scientific Continued Pretraining with Fine-grained MoE Upcyclingcited by 4
  34. 2024Performance Characterization of Expert Router for Scalable LLM Inferencecited by 2
  35. 2025Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectivescited by 2
  36. 2025A Taxonomy of Transcendencecited by 2
  37. 2026Optimal Splitting of Language Models from Mixtures to Specialized Domainscited by 1
  38. 2025ORXE: Orchestrating Experts for Dynamically Configurable Efficiencycited by 0

50 most cited of 60 All 60 (JSON)