mlsanityData to 7 Oct 2026

LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Training

arXiv 2411.15708 ↗First posted 24 Nov 2024cs.CLcited by 25

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Recently, inspired by the concept of sparsity, Mixture-of-Experts (MoE) models have gained increasing popularity for scaling model size while keeping the number of activated parameters constant. In this study, we thoroughly investigate the sparsity of the dense LLaMA model by constructing MoE for both the attention (i.e., Attention MoE) and MLP (i.e., MLP MoE) modules in the transformer blocks. Specifically, we investigate different expert construction methods and granularities under the same activation conditions to analyze the impact of sparsifying the model. Additionally, to comprehensively evaluate the model's capabilities across various domains (e.g., conversation, code, math) after sparsification, we apply sparsity to the instructed large language models (LLMs) and construct instructed MoE models. To counteract the performance degradation resulting from increased sparsity, we design a two-stage post-training strategy to enhance model performance. Experiments on the LLaMA3 model demonstrate the potential effectiveness of this approach for future developments of instructed MoE models. The source codes and models are available at: urlhttps://github.com/OpenSparseLLMs/LLaMA-MoE-v2.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 29 of its 37 references that are in the map. First posted on arXiv 24 Nov 2024.

Builds on

29 of 37 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free14 Oct
  2. 2024Upcycling Large Language Models into Mixture of Experts10 Oct
  3. 2024OLMoE: Open Mixture-of-Experts Language Models3 Sep
  4. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training24 Jun
  5. 2024Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Experts17 Jun
  6. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models3 Jun
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2023LIMA: Less Is More for Alignmentcited by 1,080
8 more
  1. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  2. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  3. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  4. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  5. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  6. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  7. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 114
  8. 2024CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcyclingcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 17; 2026: 8.

25 papers.

Sort:

By area:

  1. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  2. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  3. 2025MoM: Linear Sequence Modeling with Mixture-of-Memoriescited by 44
  4. 2025Liger: Linearizing Large Language Models to Gated Recurrent Structurescited by 32
  5. 2025Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignmentcited by 22
  6. 2025ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruningcited by 10
  7. 2025Native Hybrid Attention for Efficient Sequence Modelingcited by 10
  8. 2025MoE-DP: An MoE-Enhanced Diffusion Policy for Robust Long-Horizon Robotic Manipulation with Skill Decomposition and Failure Recoverycited by 9
  9. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  10. 2025Analytical FFN-to-MoE Restructuring via Activation Pattern Analysiscited by 7
  11. 2025Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Expertscited by 6
  12. 2025UMoE: Unifying Attention and FFN with Shared Expertscited by 6
13 more
  1. 2025SEE: Continual Fine-tuning with Sequential Ensemble of Expertscited by 4
  2. 2026SegMoTE: Token-Level Mixture of Experts for Medical Image Segmentationcited by 2
  3. 2026Grouped Query Experts: Mixture-of-Experts on GQA Self-Attentioncited by 2
  4. 2025A Retrospective Systematic Study on Hierarchical Sparse Query Transformer-assisted Ultrasound Screening for Early Hepatocellular Carcinomacited by 1
  5. 2025MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designscited by 1
  6. 2025StyleQoRA: Quality-Aware Low-Rank Adaptation for Few-Shot Multi-Style Editingcited by 1
  7. 2026MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inferencecited by 1
  8. 2025Sparse Mixture-of-Experts for Multi-Channel Imaging: Are All Channel Interactions Required?cited by 0
  9. 2026ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patternscited by 0
  10. 2026Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEscited by 0
  11. 2026DOT-MoE: Differentiable Optimal Transport for MoEficationcited by 0
  12. 2026Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactionscited by 0
  13. 2026When Routing Reveals Membership: Privacy Leakage from MoE Router Telemetrycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution
  2. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
  3. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
  4. 2025Sigma-MoE-Tiny Technical Report
  5. 2024A Survey on Mixture of Experts in Large Language Models
  6. 2024Scaling Laws for Fine-Grained Mixture of Experts
  7. 2025Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks
  8. 2025dots.llm1 Technical Report