mlsanityData to 7 Oct 2026

Sparse Mixture-of-Experts are Domain Generalizable Learners

arXiv 2206.04046 ↗First posted 8 Jun 2022cs.CV cs.AI cs.LGcited by 85

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Human visual perception can easily generalize to out-of-distributed visual data, which is far beyond the capability of modern machine learning models. Domain generalization (DG) aims to close this gap, with existing DG methods mainly focusing on the loss function design. In this paper, we propose to explore an orthogonal direction, i.e., the design of the backbone architecture. It is motivated by an empirical finding that transformer-based models trained with empirical risk minimization (ERM) outperform CNN-based models employing state-of-the-art (SOTA) DG algorithms on multiple DG datasets. We develop a formal framework to characterize a network's robustness to distribution shifts by studying its architecture's alignment with the correlations in the dataset. This analysis guides us to propose a novel DG model built upon vision transformers, namely Generalizable Mixture-of-Experts (GMoE). Extensive experiments on DomainBed demonstrate that GMoE trained with ERM outperforms SOTA DG baselines by a large margin. Moreover, GMoE is complementary to existing DG methods and its performance is substantially improved when trained with DG algorithms.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 72 of its 103 references that are in the map. First posted on arXiv 8 Jun 2022.

Builds on

72 of 103 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Scaling Vision with Sparse Mixture of Experts10 Jun

Other references · 71

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2017Squeeze-and-Excitation Networkscited by 4,763
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  11. 2019Invariant Risk Minimizationcited by 1,832
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
38 more
  1. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  2. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  3. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  4. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  5. 2017Random Erasing Data Augmentationcited by 1,235
  6. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,194
  7. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
  8. 2020In Search of Lost Domain Generalizationcited by 1,042
  9. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
  10. 2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 762
  11. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
  12. 2018Recognition in Terra Incognitacited by 655
  13. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  14. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  15. 2019Domain Generalization via Model-Agnostic Learning of Semantic Featurescited by 405
  16. 2017Functional Map of the Worldcited by 338
  17. 2022How Do Vision Transformers Work?cited by 305
  18. 2020Learning to Learn Single Domain Generalizationcited by 295
  19. 2021Gradient Matching for Domain Generalizationcited by 258
  20. 2019Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain Datacited by 245
  21. 2021TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentationcited by 225
  22. 2021Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalizationcited by 220
  23. 2020How Neural Networks Extrapolate: From Feedforward to Graph Neural Networkscited by 218
  24. 2019Fixing the train-test resolution discrepancycited by 209
  25. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  26. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  27. 2021Partial success in closing the gap between human and machine visioncited by 185
  28. 2020Adaptive Risk Minimization: Learning to Adapt to Domain Shiftcited by 183
  29. 2014Object Detectors Emerge in Deep Scene CNNscited by 178
  30. 2021A Fine-Grained Analysis on Distribution Shiftcited by 171
  31. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  32. 2019What Can Neural Networks Reason About?cited by 155
  33. 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
  34. 2021Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalizationcited by 144
  35. 2022Domain Generalization by Mutual-Information Regularization with Pre-trained Modelscited by 133
  36. 2019Feature-Critic Networks for Heterogeneous Domain Generalizationcited by 130
  37. 2020Batch Value-function Approximation with Only Realizabilitycited by 103
  38. 2020Neural Networks Fail to Learn Periodic Functions and How to Fix Itcited by 97

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 25; 2025: 26; 2026: 16.

85 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  2. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 67
  3. 2023Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modelingcited by 63
  4. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  5. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 55
  6. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 36
  7. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Expertscited by 36
  8. 2024Adaptive Multi-Scale Decomposition Framework for Time Series Forecastingcited by 34
  9. 2023Robust Mixture-of-Expert Training for Convolutional Neural Networkscited by 30
  10. 2023Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Expertscited by 28
  11. 2024MoE-FFD: Mixture of Experts for Generalized and Parameter-Efficient Face Forgery Detectioncited by 27
  12. 2024On Least Square Estimation in Softmax Gating Mixture of Expertscited by 23
38 more
  1. 2024Routers in Vision Mixture of Experts: An Empirical Studycited by 22
  2. 2024SM3Det: A Unified Model for Multi-Modal Remote Sensing Object Detectioncited by 22
  3. 2024DGMamba: Domain Generalization via Generalized State Space Modelcited by 21
  4. 2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21
  5. 2023Backdoor Attack with Sparse and Invisible Triggercited by 20
  6. 2023GraphMETRO: Mitigating Complex Graph Distribution Shifts via Mixture of Aligned Expertscited by 19
  7. 2024Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Expertscited by 19
  8. 2023Unlocking Emergent Modularity in Large Language Modelscited by 17
  9. 2023Rethinking Domain Generalization: Discriminability and Generalizabilitycited by 16
  10. 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Expertscited by 16
  11. 2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 16
  12. 2024DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Modelscited by 15
  13. 2024Learn to Preserve and Diversify: Parameter-Efficient Group with Orthogonal Regularization for Domain Generalizationcited by 14
  14. 2023Domain Generalization via Nuclear Norm Regularizationcited by 12
  15. 2023Domain Generalization Using Large Pretrained Models with Mixture-of-Adapterscited by 11
  16. 2023Diverse Target and Contribution Scheduling for Domain Generalizationcited by 10
  17. 2024Conditional computation in neural networks: principles and research trendscited by 10
  18. 2025DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Modelscited by 10
  19. 2024MomentumSMoE: Integrating Momentum into Sparse Mixture of Expertscited by 9
  20. 2024Hierarchical Mixture of Experts: Generalizable Learning for High-Level Synthesiscited by 9
  21. 2024HCVP: Leveraging Hierarchical Contrastive Visual Prompt for Domain Generalizationcited by 8
  22. 2023A Theoretical Explanation of Activation Sparsity through Flat Minima and Adversarial Robustnesscited by 7
  23. 2025Learning Fine-grained Domain Generalization via Hyperbolic State Space Hallucinationcited by 6
  24. 2025Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Modelcited by 6
  25. 2023An Empirical Study of Pre-trained Model Selection for Out-of-Distribution Generalization and Calibrationcited by 5
  26. 2023Towards Unified and Effective Domain Generalizationcited by 5
  27. 2025Equipping Vision Foundation Model with Mixture of Experts for Out-of-Distribution Detectioncited by 5
  28. 2024START: A Generalized State Space Model with Saliency-Driven Token-Aware Transformationcited by 4
  29. 2025Correlative and Discriminative Label Grouping for Multi-Label Visual Prompt Tuningcited by 4
  30. 2025How Many Experts Are Enough? Towards Optimal Semantic Specialization for Mixture-of-Expertscited by 4
  31. 2024CLIP-Mamba: CLIP Pretrained Mamba Models with OOD and Hessian Evaluationcited by 3
  32. 2025Convergence Rates for Softmax Gating Mixture of Expertscited by 3
  33. 2025Domain Generalization via Discrete Codebook Learningcited by 3
  34. 2025Bias and Generalizability of Foundation Models across Datasets in Breast Mammographycited by 3
  35. 2025Generalizable Multispectral Land Cover Classification via Frequency-Aware Mixture of Low-Rank Token Expertscited by 3
  36. 2025RoME: Domain-Robust Mixture-of-Experts for MILP Solution Prediction across Domainscited by 3
  37. 2023Mixture-of-Experts for Open Set Domain Adaptation: A Dual-Space Detection Approachcited by 2
  38. 2024Graph Knowledge Distillation to Mixture of Expertscited by 2

50 most cited of 85 All 85 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Mixture-of-Experts Models in Vision: Routing, Optimization, and Generalization
  2. 2024PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalization
  3. 2025Generalizing Vision-Language Models with Dedicated Prompt Guidance
  4. 2024Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study
  5. 2025Mixture of Group Experts for Learning Invariant Representations
  6. 2022HMOE: Hypernetwork-based Mixture of Experts for Domain Generalization
  7. 2023Strong but simple: A Baseline for Domain Generalized Dense Perception by CLIP-based Transfer Learning
  8. 2024Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsity