mlsanityData to 7 Oct 2026

Mixture-of-Transformers Learn Faster: A Theoretical Study on Classification Problems

arXiv 2510.27004 ↗First posted 30 Oct 2025cs.LGcited by 0

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Mixture-of-Experts (MoE) models improve transformer efficiency but lack a unified theoretical explanation, especially when both feed-forward and attention layers are allowed to specialize. To this end, we study the Mixture-of-Transformers (MoT), a tractable theoretical framework in which each transformer block acts as an expert governed by a continuously trained gating network. This design allows us to isolate and study the core learning dynamics of expert specialization and attention alignment. In particular, we develop a three-stage training algorithm with continuous training of the gating network, and show that each transformer expert specializes in a distinct class of tasks and that the gating network accurately routes data samples to the correct expert. Our analysis shows how expert specialization reduces gradient conflicts and makes each subtask strongly convex. We prove that the training drives the expected prediction loss to near zero in O(log(ε-1)) iteration steps, significantly improving over the O(ε-1) rate for a single transformer. We further validate our theoretical findings through extensive real-data experiments, demonstrating the practical effectiveness of MoT. Together, these results offer the first unified theoretical account of transformer-level specialization and learning dynamics, providing practical guidance for designing efficient large-scale models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 36 of its 45 references that are in the map. First posted on arXiv 30 Oct 2025.

Builds on

36 of 45 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention13 Dec
  2. 2022Mixture of Attention Heads: Selecting Attention Heads Per Token11 Oct
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  11. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  12. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
21 more
  1. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  2. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  3. 2020A Survey on Visual Transformercited by 653
  4. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  5. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  6. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  7. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  8. 2025MMTEB: Massive Multilingual Text Embedding Benchmarkcited by 120
  9. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 114
  10. 2023In-Context Convergence of Transformerscited by 112
  11. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  12. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 85
  13. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  14. 2023Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Taskscited by 72
  15. 2023A Review of Deep Learning Techniques for Speech Processingcited by 52
  16. 2024Theory on Mixture-of-Experts in Continual Learningcited by 50
  17. 2023Transformers in Speech Processing: A Surveycited by 38
  18. 2024How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?cited by 35
  19. 2020A Mixture of h-1 Heads is Better than h Headscited by 22
  20. 2024Training Dynamics of Transformers to Recognize Word Co-occurrence via Gradient Flow Analysiscited by 8
  21. 2024Speech Emotion Recognition Via CNN-Transformer and Multidimensional Attention Mechanismcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021EvoMoE: An Evolutional Mixture-of-Experts Training Framework via Dense-To-Sparse Gate
  2. 2026Generalization and Scaling Laws for Mixture-of-Experts Transformers
  3. 2026Mixture of Chapters: Scaling Learnt Memory in Transformers
  4. 2024Improving the Downstream Performance of Mixture-of-Experts Transformers via Weak Vanilla Transformers
  5. 2026Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformers
  6. 2026Synergistic Intra- and Cross-Layer Regularization Losses for MoE Expert Specialization
  7. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models
  8. 2024How Lightweight Can A Vision Transformer Be