mlsanityData to 7 Oct 2026

UMoE: Unifying Attention and FFN with Shared Experts

arXiv 2505.07260 ↗First posted 12 May 2025cs.LG cs.AIcited by 6

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Sparse Mixture of Experts (MoE) architectures have emerged as a promising approach for scaling Transformer models. While initial works primarily incorporated MoE into feed-forward network (FFN) layers, recent studies have explored extending the MoE paradigm to attention layers to enhance model performance. However, existing attention-based MoE layers require specialized implementations and demonstrate suboptimal performance compared to their FFN-based counterparts. In this paper, we aim to unify MoE designs in attention and FFN layers by introducing a novel reformulation of the attention mechanism, that reveals an underlying FFN-like structure within attention modules. Our proposed architecture, UMoE, achieves superior performance through attention-based MoE layers while enabling efficient parameter sharing between FFN and attention components.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 32 of its 36 references that are in the map. First posted on arXiv 12 May 2025.

Builds on

32 of 36 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024MoH: Multi-Head Attention as Mixture-of-Head Attention15 Oct
  2. 2024MoEUT: Mixture-of-Experts Universal Transformers25 May
  3. 2023SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention13 Dec
  4. 2022Mixture of Attention Heads: Selecting Attention Heads Per Token11 Oct
  5. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  10. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  11. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  12. 2018Universal Transformerscited by 613
14 more
  1. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  2. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  3. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  4. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  5. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  6. 2021Hash Layers For Large Sparse Modelscited by 236
  7. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  8. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
  9. 2023Scaling Laws for Sparsely-Connected Foundation Modelscited by 41
  10. 2023Neuron-Level Knowledge Attribution in Large Language Modelscited by 37
  11. 2024Multi-Head Mixture-of-Expertscited by 30
  12. 2024XMoE: Sparse Models with Fine-grained and Adaptive Expert Selectioncited by 27
  13. 2024LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Trainingcited by 25
  14. 2023Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Modelcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Point-MoE: Large-Scale Multi-Dataset Training with Mixture-of-Experts for 3D Semantic Segmentationcited by 3
  2. 2025Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Modelscited by 3
  3. 2026A2-Edit: Precise Reference-Guided Image Editing of Arbitrary Objects and Ambiguous Maskscited by 2
  4. 2026Repetition Mismatch: Why Data Mixture Experiments Don't Scale and How to Fix Themcited by 0
  5. 2026Scaling Parameter and Context in Attention: Native Sparse Attention from Mixture-of-Headcited by 0
  6. 2026On the Necessity of Attention-FFN Split in Vision Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimal Scaling Laws for Efficiency Gains in a Theoretical Transformer-Augmented Sectional MoE Framework
  2. 2026Mixture of Layers with Hybrid Attention
  3. 2026Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformers
  4. 2025Mixture-of-Transformers Learn Faster: A Theoretical Study on Classification Problems
  5. 2024Mixture of A Million Experts
  6. 2026Complete-muE: Optimal Hyperparameter Transfer and Scaling for MoE Models
  7. 2025Optimizing MoE Routers: Design, Implementation, and Evaluation in Transformer Models
  8. 2024How Lightweight Can A Vision Transformer Be