mlsanityData to 14 Oct 2026

GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanism

arXiv 2501.07890 ↗First posted 14 Jan 2025cs.CL cs.AIcited by 7

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

also in 1 more
Abstract (arXiv)

Traditional Mixture-of-Experts (MoE) networks benefit from utilizing multiple smaller expert models as opposed to a single large network. However, these experts typically operate independently, leaving a question open about whether interconnecting these models could enhance the performance of MoE networks. In response, we introduce GRAPHMOE, a novel method aimed at augmenting the cognitive depth of language models via a self-rethinking mechanism constructed on Pseudo GraphMoE networks. GRAPHMOE employs a recurrent routing strategy to simulate iterative thinking steps, thereby facilitating the flow of information among expert nodes. We implement the GRAPHMOE architecture using Low-Rank Adaptation techniques (LoRA) and conduct extensive experiments on various benchmark datasets. The experimental results reveal that GRAPHMOE outperforms other LoRA based models, achieving state-of-the-art (SOTA) performance. Additionally, this study explores a novel recurrent routing strategy that may inspire further advancements in enhancing the reasoning capabilities of language models.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 37 of its 59 references that are in the map. First posted on arXiv 14 Jan 2025.

Builds on

37 of 59 references.

References · 37

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  7. 2024Mixtral of Expertscited by 1,823
  8. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  9. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  10. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  11. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  12. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
25 more
  1. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  2. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  3. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  4. 2024Continual Learning for Large Language Models: A Surveycited by 147
  5. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 147
  6. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 113
  7. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  8. 2024Higher Layers Need More LoRA Expertscited by 102
  9. 2023Soft Merging of Experts with Adaptive Routingcited by 90
  10. 2024Mixture of A Million Expertscited by 72
  11. 2024Attention Heads of Large Language Models: A Surveycited by 67
  12. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollarscited by 57
  13. 2024MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Modelscited by 56
  14. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 50
  15. 2024Mixture-of-Subspaces in Low-Rank Adaptationcited by 47
  16. 2024Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengthscited by 44
  17. 2019SocialIQA: Commonsense Reasoning about Social Interactionscited by 37
  18. 2024Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-trainingcited by 37
  19. 2024xFinder: Large Language Models as Automated Evaluators for Reliable Evaluationcited by 19
  20. 2024Yuan 2.0-M32: Mixture of Experts with Attention Routercited by 18
  21. 2024SLIDE: A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluationcited by 11
  22. 2024Improving Transformers with Dynamically Composable Multi-Head Attentioncited by 10
  23. 2024X-ray Made Simple: Lay Radiology Report Generation and Robust Evaluationcited by 10
  24. 2023AGaLiTe: Approximate Gated Linear Transformers for Online Reinforcement Learningcited by 6
  25. 2023From Complex to Simple: Unraveling the Cognitive Tree for Reasoning with Small Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structurescited by 3
  2. 2025KCS: Diversify Multi-hop Question Generation with Knowledge Composition Samplingcited by 2
  3. 2025You Need Reasoning to Learn Reasoning: The Limitations of Label-Free RL in Weak Base Modelscited by 2
  4. 2026HiEdit: Lifelong Model Editing with Hierarchical Reinforcement Learningcited by 1
  5. 2026SDG-MoE: Signed Debate Graph Mixture-of-Expertscited by 1
  6. 2026Out-of-Distribution Generalization in Graph Foundation Modelscited by 0
  7. 2026KA2L: A Knowledge-Aware Active Learning Framework for LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts
  2. 2026Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns
  3. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  4. 2025ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Experts
  5. 2025Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Say
  6. 2024LaDiMo: Layer-wise Distillation Inspired MoEfier
  7. 2025Mixture of Experts in Large Language Models
  8. 2025GraphOmni: A Comprehensive and Extensible Benchmark Framework for Large Language Models on Graph-theoretic Tasks