mlsanityData to 7 Oct 2026

Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structures

arXiv 2510.16411 ↗First posted 18 Oct 2025cs.LG cs.AIcited by 3

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Sparse Mixture of Experts (SMoE) has emerged as a promising solution to achieving unparalleled scalability in deep learning by decoupling model parameter count from computational cost. By activating only a small subset of parameters per sample, SMoE enables significant growth in model capacity while maintaining efficiency. However, SMoE struggles to adapt to distributional shifts, leading to reduced robustness under data contamination. In this work, we introduce SymphonySMoE, a novel family of SMoE that introduces a social graph to model interactions among experts. This graph-based structure enhances the token routing process, addressing the robustness challenges that are inherent in conventional SMoE designs. SymphonySMoE is lightweight, modular, and integrates seamlessly with existing SMoE-based models such as the XMoE and the Generalist Language Model. We provide both theoretical analysis and empirical evidence demonstrating SymphonySMoE's advantages over baseline SMoE. Extensive experiments on language modeling and visual instruction tuning validate our method's effectiveness. We further highlight the scalability of SymphonySMoE to models with 4.2 and 7.4 billion parameters, showcasing its applicability in fine-tuning tasks for large-scale systems.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 63 of its 78 references that are in the map. First posted on arXiv 18 Oct 2025.

Builds on

63 of 78 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanism14 Jan
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  3. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 57

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
38 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2016Pointer Sentinel Mixture Modelscited by 2,992
  3. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  12. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  13. 2019Natural Adversarial Examplescited by 1,401
  14. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  15. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  16. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  17. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  18. 2020Graph Neural Networks in Recommender Systems: A Surveycited by 493
  19. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  20. 2023Scaling Data-Constrained Language Modelscited by 391
  21. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  22. 2023From Sparse to Soft Mixtures of Expertscited by 245
  23. 2021Hash Layers For Large Sparse Modelscited by 236
  24. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  25. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  26. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  27. 2022Understanding The Robustness in Vision Transformerscited by 144
  28. 2023Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behaviorcited by 128
  29. 2023In-Context Operator Learning with Data Prompts for Differential Equation Problemscited by 98
  30. 2022A Survey of Graph Neural Networks for Social Recommender Systemscited by 65
  31. 2023Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modelingcited by 63
  32. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  33. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  34. 2019Improving Transformer Models by Reordering their Sublayerscited by 52
  35. 2022Sparsity-Constrained Optimal Transportcited by 32
  36. 2023Brainformers: Trading Simplicity for Efficiencycited by 31
  37. 2023Robust Mixture-of-Expert Training for Convolutional Neural Networkscited by 30
  38. 2022On the Adversarial Robustness of Mixture of Expertscited by 27

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026SDG-MoE: Signed Debate Graph Mixture-of-Expertscited by 1
  2. 2026Higher-order pruning of experts in mixture-of-experts language modelscited by 1
  3. 2026Router Prior Bias: Preserving Base Routing Structure in MoE Post-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Sparse Mixture of Experts from a Unified Perspective
  2. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs
  3. 2025S2MoE: Robust Sparse Mixture of Experts via Stochastic Learning
  4. 2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models
  5. 2024GMoE: Empowering LLMs Fine-Tuning via MoE Graph Collaboration
  6. 2024MomentumSMoE: Integrating Momentum into Sparse Mixture of Experts
  7. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  8. 2026Eigenvectors of Experts are Training-free Non-collapsing Routers