Mixture of Layers with Hybrid Attention
On the map: Mixture-of-experts → Expert routing strategies
Abstract (arXiv)
Standard Mixture-of-Experts (MoE) transformers route tokens to expert subnetworks within each layer, but the layer structure itself remains monolithic. We introduce Mixture of Layers (MoL), which replaces full-width transformer blocks (d_model) with K parallel thin blocks at reduced dimensionality (d_thin << d_model), connected via learned down/up projections and composed via top-k block routing. Scaling sparse block routing to many blocks creates an attention coverage problem, as each block sees fewer tokens. We address this by introducing hybrid attention, which pairs one shared softmax block for global context with Gated DeltaNet linear attention in routed blocks.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 25 of its 31 references that are in the map. First posted on arXiv 10 May 2026.
Builds on
25 of 31 references.
Closest ancestors · 5
Same direction, newest first
- 2026Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality15 Apr
- 2026Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation5 Mar
- 2025Union of Experts: Adapting Hierarchical Routing to Equivalently Decomposed Transformer4 Mar
- 2024MoEUT: Mixture-of-Experts Universal Transformers25 May
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan
Other references · 20
- 2017Squeeze-and-Excitation Networkscited by 4,763
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2016Pointer Sentinel Mixture Modelscited by 2,992
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
- 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
- 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
8 more
- 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
- 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
- 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
- 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
- 2023Alternating Updates for Efficient Transformerscited by 13
- 2024Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modulescited by 11
- 2025MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcyclingcited by 5
- 2025Flash Multi-Head Feed-Forward Networkcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2022Mixture of Attention Heads: Selecting Attention Heads Per Token
- 2026Mixture-of-Top-k Attention: Efficient Attention via Scalable Fast Weights
- 2026Design and Behavior of Sparse Mixture-of-Experts Layers in CNN-based Semantic Segmentation
- 2026Sparse Mixture-of-Experts Routing in Visual Diffusion Transformers:Diagnosis, Boundary Calibration and Evolutionary Roadmap from Routing Collapse to Selective Deadlock
- 2026Attention-Aware Routing: Coupling Routing and Attention in MoEs
- 2026Dense Mixture-of-Experts as a Reparameterized Wide FFN: A Granularity Sweep at Fixed Compute
- 2026Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers
- 2025UMoE: Unifying Attention and FFN with Shared Experts