mlsanityData to 7 Oct 2026

Improving Minimax Estimation Rates for Contaminated Mixture of Multinomial Logistic Experts via Expert Heterogeneity

arXiv 2602.00939 ↗First posted 31 Jan 2026math.ST cs.LG stat.THcited by 0

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Contaminated mixture of experts (MoE) is motivated by transfer learning methods where a pre-trained model, acting as a frozen expert, is integrated with an adapter model, functioning as a trainable expert, in order to learn a new task. Despite recent efforts to analyze the convergence behavior of parameter estimation in this model, there are still two unresolved problems in the literature. First, the contaminated MoE model has been studied solely in regression settings, while its theoretical foundation in classification settings remains absent. Second, previous works on MoE models for classification capture pointwise convergence rates for parameter estimation without any guaranty of minimax optimality. In this work, we close these gaps by performing, for the first time, the convergence analysis of a contaminated mixture of multinomial logistic experts with homogeneous and heterogeneous structures, respectively. In each regime, we characterize uniform convergence rates for estimating parameters under challenging settings where ground-truth parameters vary with the sample size. Furthermore, we also establish corresponding minimax lower bounds to ensure that these rates are minimax optimal. Notably, our theories offer an important insight into the design of contaminated MoE, that is, expert heterogeneity yields faster parameter estimation rates and, therefore, is more sample-efficient than expert homogeneity.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 24 of its 42 references that are in the map. First posted on arXiv 31 Jan 2026.

Builds on

24 of 42 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts24 May
  2. 2024Understanding Expert Structures on Minimax Parameter Estimation in Contaminated Mixture of Experts16 Oct
  3. 2024On Parameter Estimation in Deviated Gaussian Mixture of Experts7 Feb
  4. 2023A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts22 Oct
  5. 2023Demystifying Softmax Gating Function in Gaussian Mixture of Experts5 May

Other references · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  8. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  9. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  10. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
  11. 2024Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Expertscited by 69
  12. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 67
7 more
  1. 2024HMoE: Heterogeneous Mixture of Experts for Language Modelingcited by 28
  2. 2022A Mixture-of-Expert Approach to RL-based Dialogue Managementcited by 16
  3. 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Expertscited by 16
  4. 2024MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routingcited by 16
  5. 2025Heterogeneous Mixture of Experts for Remote Sensing Image Super-Resolutioncited by 11
  6. 2024MoE-CT: A Novel Approach For Large Language Models Training With Resistance To Catastrophic Forgettingcited by 6
  7. 2025HDEE: Heterogeneous Domain Expert Ensemblecited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2019Regularized Estimation and Feature Selection in Mixtures of Generalized Linear Experts
  2. 2017An Introduction to the Practical and Theoretical Aspects of Mixture-of-Experts Modeling
  3. 2018Regularized Maximum Likelihood Estimation and Feature Selection in Mixtures-of-Experts Models
  4. 2023Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts
  5. 2024Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study
  6. 2024Learning Mixtures of Experts with EM: A Mirror Descent Perspective
  7. 2025Mixture of Many Zero-Compute Experts: A High-Rate Quantization Theory Perspective
  8. 2024On Least Square Estimation in Softmax Gating Mixture of Experts