mlsanityData to 14 Oct 2026

Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models

arXiv 2606.16825 ↗First posted 15 Jun 2026cs.CL cs.AI cs.LGcited by 4

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-Experts (MoE) architectures efficiently scale Large Language Models (LLMs) by activating only a small fraction of their experts per token, yet the full parameter count - dominated by the expert parameters - must be held in training and inference memory. To address this, we introduce Expert Tying, an architectural modification that shares expert parameters across consecutive transformer layers while preserving independent, layer-wise routing and attention. We evaluate this approach across common, state-of-the-art architectures, including OLMoE, Qwen3, and DeepSeek-style MoEs. Our pretraining experiments demonstrate that tying experts can reduce memory footprint by almost 2x at virtually no degradation in perplexity or downstream quality. By exploiting the parameter redundancy inherent in MoE pathways, our method provides a highly favorable compute-to-memory trade-off, advancing efficient training and scaling of next-generation LLMs.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 31 of its 47 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

31 of 47 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling3 Jun
  2. 2025ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Experts20 Oct
  3. 2025Qwen3 Technical Report14 May
  4. 2024DeepSeek-V3 Technical Report27 Dec
  5. 2024OLMoE: Open Mixture-of-Experts Language Models3 Sep
  6. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 23

  1. 2018Universal Transformerscited by 647
  2. 2025Kimi K2: Open Agentic Intelligencecited by 567
  3. 2025Muon is Scalable for LLM Trainingcited by 428
  4. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  5. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  6. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformerscited by 240
  7. 2025Scaling Latent Reasoning via Looped Language Modelscited by 165
  8. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 133
  9. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 78
  10. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 61
  11. 2024MoEUT: Mixture-of-Experts Universal Transformerscited by 59
  12. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 55
11 more
  1. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 54
  2. 2026How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Modelscited by 24
  3. 2026SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursioncited by 12
  4. 2026Hyperloop Transformerscited by 12
  5. 2025MeSH: Memory-as-State-Highways for Recursive Transformerscited by 8
  6. 2026Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformationcited by 8
  7. 2025RingFormer: Rethinking Recurrent Transformer with Adaptive Level Signalscited by 6
  8. 2026Synergistic Intra- and Cross-Layer Regularization Losses for MoE Expert Specializationcited by 5
  9. 2026Path-Constrained Mixture-of-Expertscited by 4
  10. 2025Megrez2 Technical Reportcited by 2
  11. 2025Improving Recursive Transformers with Mixture of LoRAscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026SMELT: Scaling Laws for Compute-Matched MoE Looped Transformerscited by 18
  2. 2026How to Loop MoE: Flatten the Experts, Untie the Attentioncited by 1
  3. 2026Universal Test-Time Trainingcited by 0
  4. 2026LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Unifying Mixture of Experts and Multi-Head Latent Attention for Efficient Language Models
  2. 2025Advancing Expert Specialization for Better MoE
  3. 2026Slicing and Dicing: Configuring Optimal Mixtures of Experts
  4. 2025Mixture of Experts in Large Language Models
  5. 2026MoRE: Mixture of Reused Experts
  6. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
  7. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  8. 2025FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models