mlsanityData to 14 Oct 2026

Soft Merging of Experts with Adaptive Routing

arXiv 2306.03745 ↗First posted 6 Jun 2023cs.LGcited by 90

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Sparsely activated neural networks with conditional computation learn to route their inputs through different "expert" subnetworks, providing a form of modularity that densely activated models lack. Despite their possible benefits, models with learned routing often underperform their parameter-matched densely activated counterparts as well as models that use non-learned heuristic routing strategies. In this paper, we hypothesize that these shortcomings stem from the gradient estimation techniques used to train sparsely activated models that use non-differentiable discrete routing decisions. To address this issue, we introduce Soft Merging of Experts with Adaptive Routing (SMEAR), which avoids discrete routing by using a single "merged" expert constructed via a weighted average of all of the experts' parameters. By routing activations through a single merged expert, SMEAR does not incur a significant increase in computational costs and enables standard gradient-based training. We empirically validate that models using SMEAR outperform models that route based on metadata or learn sparse routing through gradient estimation. Furthermore, we provide qualitative analysis demonstrating that the experts learned via SMEAR exhibit a significant amount of specialization. All of the code used in our experiments is publicly available.

Key facts. Cited by 90 papers in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 94 of its 112 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

94 of 112 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022On the Representation Collapse of Sparse Mixture of Experts20 Apr
  2. 2022StableMoE: Stable Routing Strategy for Mixture of Experts18 Apr
  3. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  4. 2021Taming Sparsely Activated Transformer with Stochastic Experts8 Oct
  5. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Experts5 Oct
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 88

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  11. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
38 more
  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  2. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  6. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  8. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  9. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  10. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  11. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  12. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  13. 2018Neural Network Acceptability Judgmentscited by 958
  14. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  15. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  16. 2021Robust fine-tuning of zero-shot modelscited by 797
  17. 2019How multilingual is Multilingual BERT?cited by 645
  18. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  19. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  20. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  21. 2017Learning multiple visual domains with residual adapterscited by 557
  22. 2021Editing Factual Knowledge in Language Modelscited by 553
  23. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
  24. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  25. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  26. 2021Dynamic Neural Networks: A Surveycited by 408
  27. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  28. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
  29. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  30. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
  31. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 289
  32. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
  33. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
  34. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 255
  35. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  36. 2021Hash Layers For Large Sparse Modelscited by 237
  37. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 226
  38. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 215

50 most cited of 88 All 88 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 28; 2025: 33; 2026: 21.

90 papers.

Sort:

By area:

  1. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  2. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  3. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  4. 2023From Sparse to Soft Mixtures of Expertscited by 248
  5. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  6. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 150
  7. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  8. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  9. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  10. 2024LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wildcited by 68
  11. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  12. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 54
38 more
  1. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  2. 2024MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selectioncited by 40
  3. 2024Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-trainingcited by 37
  4. 2024LLM Augmented LLMs: Expanding Capabilities through Compositioncited by 35
  5. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 35
  6. 2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Mergingcited by 23
  7. 2024DAM: Dynamic Adapter Merging for Continual Video QA Learningcited by 19
  8. 2024Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learningcited by 14
  9. 2025Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learningcited by 13
  10. 2023Sweeping Heterogeneity with Smart MoPs: Mixture of Prompts for LLM Task Adaptationcited by 12
  11. 2024Realistic Evaluation of Model Merging for Compositional Generalizationcited by 12
  12. 2024Conditional computation in neural networks: principles and research trendscited by 10
  13. 2025Revisiting Model Interpolation for Efficient Reasoningcited by 10
  14. 2026REAM: Merging Improves Pruning of Experts in LLMscited by 10
  15. 2024MoE Jetpack: From Dense Checkpoints to Adaptive Mixture of Experts for Vision Taskscited by 9
  16. 2024Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruningcited by 8
  17. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  18. 2025Dynamic Mixture of Progressive Parameter-Efficient Expert Library for Lifelong Robot Learningcited by 8
  19. 2025GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanismcited by 7
  20. 2023Adaptive Computation Modules: Granular Conditional Computation For Efficient Inferencecited by 6
  21. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
  22. 2025BTS: Harmonizing Specialized Experts into a Generalist LLMcited by 6
  23. 2025Efficiently Editing Mixture-of-Experts Models with Compressed Expertscited by 6
  24. 2025BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsitycited by 6
  25. 2023Mixture of Tokens: Continuous MoE through Cross-Example Aggregationcited by 5
  26. 2024Exploring the Benefit of Activation Sparsity in Pre-trainingcited by 5
  27. 20251bit-Merging: Dynamic Quantized Merging for Large Language Modelscited by 5
  28. 2025CAMEx: Curvature-aware Merging of Expertscited by 5
  29. 2025A Systematic Study of In-the-Wild Model Merging for Large Language Modelscited by 5
  30. 2024What the Weight?! A Unified Framework for Zero-Shot Knowledge Compositioncited by 4
  31. 2024Glider: Global and Local Instruction-Driven Expert Routercited by 4
  32. 2025MetricHMSR:Metric Human Mesh and Scene Recovery from Monocular Imagescited by 4
  33. 2025RouteMark: A Fingerprint for Intellectual Property Attribution in Routing-based Model Mergingcited by 4
  34. 2026SimMerge: Learning to Select Merge Operators from Similarity Signalscited by 4
  35. 2026CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Mergingcited by 4
  36. 2026SAMoE-VLA: A Scene Adaptive Mixture-of-Experts Vision-Language-Action Model for Autonomous Drivingcited by 4
  37. 2024Unleashing the Power of Meta-tuning for Few-shot Generalization Through Sparse Interpolated Expertscited by 3
  38. 2025MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 3

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Bridging Local and Global Knowledge: Cascaded Mixture-of-Experts Learning for Near-Shortest Path Routing
  2. 2026Towards a Statistical Understanding of Mixture-of-Experts
  3. 2025Selective Sinkhorn Routing for Improved Sparse Mixture of Experts
  4. 2026RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing
  5. 2026Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts
  6. 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Experts
  7. 2026How Sparse Probability Maps Shape Mixture-of-Experts Routing
  8. 2026Expert Routing for Communication-Efficient MoE via Finite Expert Banks