mlsanityData to 7 Oct 2026

Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models

arXiv 2208.03306 ↗First posted 5 Aug 2022cs.CLcited by 187

Not on the map. Why?

Cited by 39 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Branch-Train-Merge (BTM), a communication-efficient algorithm for embarrassingly parallel training of large language models (LLMs). We show it is possible to independently train subparts of a new class of LLMs on different subsets of the data, eliminating the massive multi-node synchronization currently required to train LLMs. BTM learns a set of independent expert LMs (ELMs), each specialized to a different textual domain, such as scientific or legal text. These ELMs can be added and removed to update data coverage, ensembled to generalize to new domains, or averaged to collapse back to a single LM for efficient inference. New ELMs are learned by branching from (mixtures of) ELMs in the current set, further training the parameters on data for the new domain, and then merging the resulting model back into the set for future use. Experiments show that BTM improves in- and out-of-domain perplexities as compared to GPT-style Transformer LMs, when controlling for training cost. Through extensive analysis, we show that these results are robust to different ELM initialization schemes, but require expert domain specialization; LM ensembles with random data splits do not perform well. We also present a study of scaling BTM into a new corpus of 64 domains (192B whitespace-separated tokens in total); the resulting LM (22.4B total parameters) performs as well as a Transformer LM trained with 2.5 times more compute. These gains grow with the number of domains, suggesting more aggressive parallelism could be used to efficiently train larger models in future work.

Key facts. Cited by 187 papers in the map (data through 7 Oct 2026). Builds on 53 of its 82 references that are in the map. First posted on arXiv 5 Aug 2022.

Builds on

53 of 82 references.

References · 53

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  11. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  12. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
38 more
  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  2. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  3. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  4. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  5. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  6. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  7. 2021Robust fine-tuning of zero-shot modelscited by 785
  8. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  9. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  10. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  11. 2019Defending Against Neural Fake Newscited by 553
  12. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  13. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  14. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  15. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  16. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  17. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  18. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  19. 2020The Pushshift Reddit Datasetcited by 289
  20. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  21. 2021Hash Layers For Large Sparse Modelscited by 236
  22. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
  23. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  24. 2018A Simple Method for Commonsense Reasoningcited by 223
  25. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  26. 2016Demographic Dialectal Variation in Social Media: A Case Study of African-American Englishcited by 194
  27. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
  28. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  29. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  30. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  31. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  32. 2019BillSum: A Corpus for Automatic Summarization of US Legislationcited by 117
  33. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  34. 2021Time Waits for No One! Analysis and Challenges of Temporal Misalignmentcited by 91
  35. 2021No One Representation to Rule Them All: Overlapping Features of Training Methodscited by 57
  36. 2021Efficient Hierarchical Domain Adaptation for Pretrained Language Modelscited by 32
  37. 2015Bayesian model averaging: A systematic review and conceptual classificationcited by 29
  38. 2021Characterizing English Variation across Social Media Communities with BERTcited by 20

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 40; 2024: 66; 2025: 36; 2026: 36.

187 papers.

Sort:

By area:

  1. 2022Editing Models with Task Arithmeticcited by 1,076
  2. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  3. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  4. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  5. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  6. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  7. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  8. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  9. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  10. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  11. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  12. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
38 more
  1. 2023RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Pathscited by 141
  2. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
  3. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 115
  4. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Expertscited by 110
  5. 2024Knowledge Fusion of Large Language Modelscited by 110
  6. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 106
  7. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 105
  8. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  9. 2023Deep Model Fusion: A Surveycited by 89
  10. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  11. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 86
  12. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
  13. 2023Modular Deep Learningcited by 82
  14. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  15. 2023MatFormer: Nested Transformer for Elastic Inferencecited by 79
  16. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  17. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  18. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 71
  19. 2024Learning to Decode Collaboratively with Multiple Language Modelscited by 61
  20. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  21. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  22. 2023Paloma: A Benchmark for Evaluating Language Model Fitcited by 53
  23. 2025Reinforcement Learning from Human Feedbackcited by 53
  24. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  25. 2023An Empirical Study of Multimodal Model Mergingcited by 52
  26. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  27. 2024LoRA Soups: Merging LoRAs for Practical Skill Composition Taskscited by 49
  28. 2023OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllerscited by 47
  29. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  30. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  31. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  32. 2024Flextron: Many-in-One Flexible Large Language Modelcited by 42
  33. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  34. 2023Emergent Modularity in Pre-trained Transformerscited by 37
  35. 2024Breaking the Curse of Multilinguality with Cross-lingual Expert Language Modelscited by 36
  36. 2024Mitigating Catastrophic Forgetting in Language Transfer via Model Mergingcited by 36
  37. 2023Time is Encoded in the Weights of Finetuned Language Modelscited by 31
  38. 2025MergeBench: A Benchmark for Merging Domain-Specialized LLMscited by 30

50 most cited of 187 All 187 (JSON)