mlsanityData to 14 Oct 2026

Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM

arXiv 2403.07816 ↗First posted 12 Mar 2024cs.CL cs.AIcited by 109

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

We investigate efficient methods for training Large Language Models (LLMs) to possess capabilities in multiple specialized domains, such as coding, math reasoning and world knowledge. Our method, named Branch-Train-MiX (BTX), starts from a seed model, which is branched to train experts in embarrassingly parallel fashion with high throughput and reduced communication cost. After individual experts are asynchronously trained, BTX brings together their feedforward parameters as experts in Mixture-of-Expert (MoE) layers and averages the remaining parameters, followed by an MoE-finetuning stage to learn token-level routing. BTX generalizes two special cases, the Branch-Train-Merge method, which does not have the MoE finetuning stage to learn routing, and sparse upcycling, which omits the stage of training experts asynchronously. Compared to alternative approaches, BTX achieves the best accuracy-efficiency tradeoff.

Key facts. Cited by 109 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 33 of its 40 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

33 of 40 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2021Program Synthesis with Large Language Modelscited by 3,637
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
21 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  4. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  8. 2016Progressive Neural Networkscited by 1,361
  9. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  10. 2023Llemma: An Open Language Model For Mathematicscited by 388
  11. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 372
  12. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  13. 2021Hash Layers For Large Sparse Modelscited by 237
  14. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  15. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  16. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  17. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 132
  18. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 121
  19. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 95
  20. 2014Deep learning with Elastic Averaging SGDcited by 90
  21. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 60

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 44; 2025: 32; 2026: 33.

109 papers.

Sort:

By area:

  1. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  2. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  3. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  4. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  5. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 163
  6. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 116
  7. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Expertscited by 62
  8. 2025Unlocking Efficient Long-to-Short LLM Reasoning with Model Mergingcited by 58
  9. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learningcited by 57
  10. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  11. 2024FuseChat: Knowledge Fusion of Chat Modelscited by 55
  12. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
38 more
  1. 2024Upcycling Large Language Models into Mixture of Expertscited by 49
  2. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  3. 2025MergeBench: A Benchmark for Merging Domain-Specialized LLMscited by 33
  4. 2025CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routingcited by 30
  5. 2024Meta Reasoning for Large Language Modelscited by 29
  6. 2025Cognitive Memory in Large Language Modelscited by 29
  7. 2025Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initializationcited by 24
  8. 2024Self-MoE: Towards Compositional Large Language Models with Self-Specialized Expertscited by 23
  9. 2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21
  10. 2024DAM: Dynamic Adapter Merging for Continual Video QA Learningcited by 19
  11. 2024On the Limitations of Compute Thresholds as a Governance Strategycited by 18
  12. 2026Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillationcited by 18
  13. 2024Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Expertscited by 17
  14. 2024PLeaS -- Merging Models with Permutations and Least Squarescited by 16
  15. 2024BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Expertscited by 16
  16. 2025Communicating Activations Between Language Model Agentscited by 16
  17. 2025Cost-Aware Contrastive Routing for LLMscited by 16
  18. 2025FlexOlmo: Open Language Models for Flexible Data Usecited by 15
  19. 2024AutoJailbreak: Exploring Jailbreak Attacks and Defenses through a Dependency Lenscited by 14
  20. 2024RouterRetriever: Routing over a Mixture of Expert Embedding Modelscited by 12
  21. 2025Activation-Guided Consensus Merging for Large Language Modelscited by 12
  22. 2025Don't Throw Away Your Pretrained Modelcited by 10
  23. 2024Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wildcited by 9
  24. 2025Local Mixtures of Experts: Essentially Free Test-Time Training via Model Mergingcited by 9
  25. 2024Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruningcited by 8
  26. 2024Nexus: Specialization meets Adaptability for Efficiently Training Mixture of Expertscited by 7
  27. 2024Collective Model Intelligence Requires Compatible Specializationcited by 7
  28. 2024NeKo: Cross-Modality Post-Recognition Error Correction with Tasks-Guided Mixture-of-Experts Language Modelcited by 7
  29. 2025MixNet: A Runtime Reconfigurable Optical-Electrical Fabric for Distributed Mixture-of-Experts Trainingcited by 7
  30. 2025Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsitycited by 7
  31. 2025DISCO Balances the Scales: Adaptive Domain- and Difficulty-Aware Reinforcement Learning on Imbalanced Datacited by 7
  32. 2024Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Mergingcited by 6
  33. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
  34. 2024Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyphcited by 6
  35. 2026MergePipe: A Budget-Aware Parameter Management System for Scalable LLM Mergingcited by 6
  36. 2025CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memorycited by 5
  37. 2025Continual Pre-training of MoEs: How robust is your router?cited by 5
  38. 2024Mixture of Modular Experts: Distilling Knowledge from a Multilingual Teacher into Specialized Modular Language Modelscited by 4

50 most cited of 109 All 109 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LocMoE: A Low-Overhead MoE for Large Language Model Training
  2. 2026Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models
  3. 2025Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts
  4. 2024Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design
  5. 2026Token-Level LLM Collaboration via FusionRoute
  6. 2024Ada-K Routing: Boosting the Efficiency of MoE-based LLMs
  7. 2025ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Experts
  8. 2025Advancing Expert Specialization for Better MoE