mlsanityData to 7 Oct 2026

BTS: Harmonizing Specialized Experts into a Generalist LLM

arXiv 2502.00075 ↗First posted 31 Jan 2025cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

We present Branch-Train-Stitch (BTS), an efficient and flexible training algorithm for combining independently trained large language model (LLM) experts into a single, capable generalist model. Following Li et al., we start with a single seed language model which is branched into domain-specific (e.g., coding or math) experts with continual pretraining. BTS combines experts into a generalist model using lightweight stitch layers, which are inserted between frozen experts and the seed LLM, and trained on a small datamix of the expert domains. Stitch layers enable the seed LLM to integrate representations from any number of experts during the forward pass, allowing it to generalize to new domains, despite remaining frozen. Because BTS does not alter the constituent LLMs, BTS provides a modular and flexible approach: experts can be easily removed and new experts can be added with only a small amount of training. Compared to alternative model merging approaches, BTS yields the best generalist performance on a variety of downstream tasks, retaining the specialized capabilities of each of the experts.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 34 of its 41 references that are in the map. First posted on arXiv 31 Jan 2025.

Builds on

34 of 41 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  9. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  12. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
22 more
  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  2. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  3. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  4. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  5. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  6. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  7. 2023Llemma: An Open Language Model For Mathematicscited by 385
  8. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  9. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  10. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  11. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 156
  12. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 151
  13. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 151
  14. 2024The rising costs of training frontier AI modelscited by 91
  15. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 90
  16. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  17. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  18. 2022Mixture of Attention Heads: Selecting Attention Heads Per Tokencited by 58
  19. 2024LLM Augmented LLMs: Expanding Capabilities through Compositioncited by 35
  20. 2024Revisiting Reliability in Large-Scale Machine Learning Research Clusterscited by 27
  21. 2024Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Trainingcited by 7
  22. 2024No Need to Talk: Asynchronous Mixture of Language Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025FlexOlmo: Open Language Models for Flexible Data Usecited by 14
  2. 2026Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillationcited by 12
  3. 2025Local Mixtures of Experts: Essentially Free Test-Time Training via Model Mergingcited by 9
  4. 2026Mixture of Masters: Sparse Chess Language Models with Player Routingcited by 1
  5. 2026Training-Free Dynamic Upcycling of Expert Language Modelscited by 1
  6. 2026Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Expertscited by 0