mlsanityData to 14 Oct 2026

Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization

arXiv 2402.12550 ↗First posted 19 Feb 2024cs.CV cs.LGcited by 28

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

The Mixture of Experts (MoE) paradigm provides a powerful way to decompose dense layers into smaller, modular computations often more amenable to human interpretation, debugging, and editability. However, a major challenge lies in the computational cost of scaling the number of experts high enough to achieve fine-grained specialization. In this paper, we propose the Multilinear Mixture of Experts (μMoE) layer to address this, focusing on vision models. μMoE layers enable scalable expert specialization by performing an implicit computation on prohibitively large weight tensors entirely in factorized form. Consequently, μMoEs (1) avoid the restrictively high inference-time costs of dense MoEs, yet (2) do not inherit the training issues of the popular sparse MoEs' discrete (non-differentiable) expert routing. We present both qualitative and quantitative evidence that scaling μMoE layers when fine-tuning foundation models for vision tasks leads to more specialized experts at the class-level, further enabling manual bias correction in CelebA attribute classification. Finally, we show qualitative results demonstrating the expert specialism achieved when pre-training large GPT2 and MLP-Mixer models with parameter-matched μMoE blocks at every layer, maintaining comparable accuracy. Our code is available at: https://github.com/james-oldfield/muMoE.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 42 of its 87 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

42 of 87 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023From Sparse to Soft Mixtures of Experts2 Aug
  2. 2022Sparsely-gated Mixture-of-Expert Layers for CNN Interpretability22 Apr
  3. 2022Residual Mixture of Experts20 Apr
  4. 2021Scaling Vision with Sparse Mixture of Experts10 Jun

Other references · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  7. 2016Equality of Opportunity in Supervised Learningcited by 2,007
  8. 2024Mixtral of Expertscited by 1,823
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  10. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  11. 2022Editing Models with Task Arithmeticcited by 1,120
  12. 2022Toy Models of Superpositioncited by 971
26 more
  1. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  2. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  3. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
  4. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
  5. 2020Fairness without Demographics through Adversarially Reweighted Learningcited by 251
  6. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 221
  7. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 209
  8. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 203
  9. 2017SMASH: One-Shot Model Architecture Search through HyperNetworkscited by 202
  10. 2022Patching open-vocabulary models by interpolating weightscited by 190
  11. 2016Tensor Ring Decompositioncited by 127
  12. 2021An Interpretability Illusion for BERTcited by 97
  13. 2022Distilling Model Failures as Directions in Latent Spacecited by 82
  14. 2016Deep Multi-task Representation Learning: A Tensor Factorisation Approachcited by 69
  15. 2016Ultimate tensorization: compressing convolutional and FC layers alikecited by 56
  16. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learnerscited by 51
  17. 2013Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networkscited by 44
  18. 2021Revisiting Dynamic Convolution via Matrix Decompositioncited by 35
  19. 2022Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Modelscited by 31
  20. 2021Technical Challenges for Training Fair Neural Networkscited by 20
  21. 2022Interpretable Mixture of Expertscited by 17
  22. 2023Last-Layer Fairness Fine-tuning is Simple and Effective for Neural Networkscited by 17
  23. 2020Multilinear Latent Conditioning for Generating Unseen Attribute Combinationscited by 14
  24. 2023A technical note on bilinear layers for interpretabilitycited by 12
  25. 2024Multilinear Operator Networkscited by 12
  26. 2021Quantifying Local Specialization in Deep Neural Networkscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 8; 2026: 16.

28 papers.

Sort:

By area:

  1. 2025Mixture of Experts Made Intrinsically Interpretablecited by 24
  2. 2024Monet: Mixture of Monosemantic Experts for Transformerscited by 13
  3. 2026Sparse Models, Sparse Safety: Unsafe Routes in Mixture-of-Experts LLMscited by 9
  4. 2024PatentGPT: A Large Language Model for Intellectual Propertycited by 8
  5. 2024Faster Language Models with Better Multi-Token Prediction Using Tensor Decompositioncited by 8
  6. 2026The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Levelcited by 7
  7. 2026MoE Lens -- An Expert Is All You Needcited by 6
  8. 2025AIM-Fair: Advancing Algorithmic Fairness via Selectively Fine-Tuning Biased Models with Contextual Synthetic Datacited by 5
  9. 2025Towards Interpretability Without Sacrifice: Faithful Dense Layer Decomposition with Mixture of Decoderscited by 5
  10. 2025Convergence Rates for Softmax Gating Mixture of Expertscited by 4
  11. 2026Mixture of Mini Experts: Overcoming the Linear Layer Bottleneck in Multiple Instance Learningcited by 4
  12. 2025Covariate Dependent Mixture of Bayesian Networkscited by 1
16 more
  1. 2025Mixture of Experts for Recognizing Depression from Interview and Reading Taskscited by 1
  2. 2026A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Expertscited by 1
  3. 2026Tensorion: A Tensor-Aware Generalization of the Muon Optimizercited by 1
  4. 2024Active partitioning: inverting the paradigm of active learningcited by 0
  5. 2025SciGPT: A Large Language Model for Scientific Literature Understanding and Knowledge Discoverycited by 0
  6. 2025Conversion rate prediction in online advertising: modeling techniques, performance evaluation and future directionscited by 0
  7. 2026Do Domain-specific Experts exist in MoE-based LLMs?cited by 0
  8. 2026Multi-Domain Learning with Global Expert Mappingcited by 0
  9. 2026fmxcoders: Factorized Masked Crosscoders for Cross-Layer Feature Discoverycited by 0
  10. 2026Learning Subset-Shared Invariances for Domain Generalization with Mixture-of-Expertscited by 0
  11. 2026Fairness-Aware Mixture-of-Experts via Subgroup Reweighting and Gate Regularizationcited by 0
  12. 2026CERA-MoA: Co-Evolving Routing Mechanisms with Continually Learning LLM Agentscited by 0
  13. 2026Infinite-Parameter LLMs: Generating and Adapting Weights from Live Datacited by 0
  14. 2026IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Expertscited by 0
  15. 2026C2FXNet: Coarse-to-Fine Scene Expert for Unified Object Detection across Adverse Weathercited by 0
  16. 2026Score the Update, Not the Token: Descent-Aligned Routing for Combinatorial LoRA Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mixture-of-Experts Models in Vision: Routing, Optimization, and Generalization
  2. 2024Mixture of Experts in Image Classification: What's the Sweet Spot?
  3. 2026Mixture of Experts with Soft Nearest Neighbor Loss: Resolving Expert Collapse via Representation Disentanglement
  4. 2025Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks
  5. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  6. 2025On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks
  7. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
  8. 2024ViMoE: An Empirical Study of Designing Vision Mixture-of-Experts