mlsanityData to 14 Oct 2026

On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts

arXiv 2505.18455 ↗First posted 24 May 2025stat.ML cs.LGcited by 1

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

The softmax-contaminated mixture of experts (MoE) model is deployed when a large-scale pre-trained model, which plays the role of a fixed expert, is fine-tuned for learning downstream tasks by including a new contamination part, or prompt, functioning as a new, trainable expert. Despite its popularity and relevance, the theoretical properties of the softmax-contaminated MoE have remained unexplored in the literature. In the paper, we study the convergence rates of the maximum likelihood estimator of gating and prompt parameters in order to gain insights into the statistical properties and potential challenges of fine-tuning with a new prompt. We find that the estimability of these parameters is compromised when the prompt acquires overlapping knowledge with the pre-trained model, in the sense that we make precise by formulating a novel analytic notion of distinguishability. Under distinguishability of the pre-trained and prompt models, we derive minimax optimal estimation rates for all the gating and prompt parameters. By contrast, when the distinguishability condition is violated, these estimation rates become significantly slower due to their dependence on the prompt convergence rate to the pre-trained model. Finally, we empirically corroborate our theoretical findings through several numerical experiments.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 31 of its 46 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

31 of 46 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Understanding Expert Structures on Minimax Parameter Estimation in Contaminated Mixture of Experts16 Oct
  2. 2024On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions3 Oct
  3. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts22 May
  4. 2024On Parameter Estimation in Deviated Gaussian Mixture of Experts7 Feb
  5. 2024Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?25 Jan
  6. 2023A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts22 Oct
  7. 2023Demystifying Softmax Gating Function in Gaussian Mixture of Experts5 May
  8. 2019Convergence Rates for Gaussian Mixtures of Experts9 Jul

Other references · 23

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2024Mixtral of Expertscited by 1,823
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  7. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  8. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  9. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 98
  10. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  11. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 78
  12. 2024Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Expertscited by 70
11 more
  1. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 68
  2. 2024Theory on Mixture-of-Experts in Continual Learningcited by 54
  3. 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Expertscited by 48
  4. 2024Mixture of Experts Meets Prompt-Based Continual Learningcited by 43
  5. 2019EM Converges for a Mixture of Many Linear Regressionscited by 27
  6. 2021SpeechMoE2: Mixture-of-Experts Model with Improved Routingcited by 26
  7. 2022A Mixture-of-Expert Approach to RL-based Dialogue Managementcited by 16
  8. 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Expertscited by 16
  9. 2024Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Promptscited by 15
  10. 2025Revisit Visual Prompt Tuning: The Expressiveness of Prompt Expertscited by 5
  11. 2025CompeteSMoE -- Statistically Guaranteed Mixture of Experts Training via Competitioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Improving Minimax Estimation Rates for Contaminated Mixture of Multinomial Logistic Experts via Expert Heterogeneitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Convergence Rates for Softmax Gating Mixture of Experts
  2. 2024On Least Square Estimation in Softmax Gating Mixture of Experts
  3. 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts
  4. 2026On Bayesian Softmax-Gated Mixture-of-Experts Models
  5. 2023Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts
  6. 2025Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps
  7. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Experts
  8. 2024Quadratic Gating Mixture of Experts: Statistical Insights into Self-Attention