# LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Training

arXiv: https://arxiv.org/abs/2411.15708 · HTML: https://mlsanity.com/paper/2411.15708/

## Key facts

- Cited by 25 papers in the map (data through 7 Oct 2026).
- Placed in MoE large language models, one of 12 directions in Mixture-of-experts.
- Builds on 29 of its 37 references that are in the map.
- First posted on arXiv 24 Nov 2024.

## Placed in

- Mixture-of-experts / [MoE large language models](https://mlsanity.com/moe/moe-large-language-models/)

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://mlsanity.com/paper/2201.05596/) (2022)
- [Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing](https://mlsanity.com/paper/2406.08464/) (2024)
- [OLMoE: Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2409.02060/) (2024)
- [OpenChat: Advancing Open-source Language Models with Mixed-Quality Data](https://mlsanity.com/paper/2309.11235/) (2023)
- [How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition](https://mlsanity.com/paper/2310.05492/) (2023)
- [LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training](https://mlsanity.com/paper/2406.16554/) (2024)
- [Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging](https://mlsanity.com/paper/2406.15479/) (2024)
- [Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models](https://mlsanity.com/paper/2406.06563/) (2024)
- [Upcycling Large Language Models into Mixture of Experts](https://mlsanity.com/paper/2410.07524/) (2024)
- [Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free](https://mlsanity.com/paper/2410.10814/) (2024)
- [CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcycling](https://mlsanity.com/paper/2409.19291/) (2024)
- [Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Experts](https://mlsanity.com/paper/2406.11256/) (2024)

## Built on by (most cited first, 25 of 25)

- [A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond](https://mlsanity.com/paper/2503.21614/) (2025)
- [A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications](https://mlsanity.com/paper/2503.07137/) (2025)
- [MoM: Linear Sequence Modeling with Mixture-of-Memories](https://mlsanity.com/paper/2502.13685/) (2025)
- [Liger: Linearizing Large Language Models to Gated Recurrent Structures](https://mlsanity.com/paper/2503.01496/) (2025)
- [Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment](https://mlsanity.com/paper/2502.16894/) (2025)
- [ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning](https://mlsanity.com/paper/2501.15316/) (2025)
- [Native Hybrid Attention for Efficient Sequence Modeling](https://mlsanity.com/paper/2510.07019/) (2025)
- [MoE-DP: An MoE-Enhanced Diffusion Policy for Robust Long-Horizon Robotic Manipulation with Skill Decomposition and Failure Recovery](https://mlsanity.com/paper/2511.05007/) (2025)
- [Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models](https://mlsanity.com/paper/2505.16056/) (2025)
- [Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis](https://mlsanity.com/paper/2502.04416/) (2025)
- [Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts](https://mlsanity.com/paper/2503.05447/) (2025)
- [UMoE: Unifying Attention and FFN with Shared Experts](https://mlsanity.com/paper/2505.07260/) (2025)
- [SEE: Continual Fine-tuning with Sequential Ensemble of Experts](https://mlsanity.com/paper/2504.06664/) (2025)
- [SegMoTE: Token-Level Mixture of Experts for Medical Image Segmentation](https://mlsanity.com/paper/2602.19213/) (2026)
- [Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention](https://mlsanity.com/paper/2606.20945/) (2026)
- [A Retrospective Systematic Study on Hierarchical Sparse Query Transformer-assisted Ultrasound Screening for Early Hepatocellular Carcinoma](https://mlsanity.com/paper/2502.03772/) (2025)
- [MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designs](https://mlsanity.com/paper/2510.19366/) (2025)
- [StyleQoRA: Quality-Aware Low-Rank Adaptation for Few-Shot Multi-Style Editing](https://mlsanity.com/paper/2511.11236/) (2025)
- [MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference](https://mlsanity.com/paper/2605.05225/) (2026)
- [Sparse Mixture-of-Experts for Multi-Channel Imaging: Are All Channel Interactions Required?](https://mlsanity.com/paper/2511.17400/) (2025)
- [ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns](https://mlsanity.com/paper/2602.15521/) (2026)
- [Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs](https://mlsanity.com/paper/2605.24681/) (2026)
- [DOT-MoE: Differentiable Optimal Transport for MoEfication](https://mlsanity.com/paper/2606.01666/) (2026)
- [Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions](https://mlsanity.com/paper/2608.18539/) (2026)
- [When Routing Reveals Membership: Privacy Leakage from MoE Router Telemetry](https://mlsanity.com/paper/2610.10616/) (2026)

## Abstract (arXiv)

Recently, inspired by the concept of sparsity, Mixture-of-Experts (MoE) models have gained increasing popularity for scaling model size while keeping the number of activated parameters constant. In this study, we thoroughly investigate the sparsity of the dense LLaMA model by constructing MoE for both the attention (i.e., Attention MoE) and MLP (i.e., MLP MoE) modules in the transformer blocks. Specifically, we investigate different expert construction methods and granularities under the same activation conditions to analyze the impact of sparsifying the model. Additionally, to comprehensively evaluate the model's capabilities across various domains (e.g., conversation, code, math) after sparsification, we apply sparsity to the instructed large language models (LLMs) and construct instructed MoE models. To counteract the performance degradation resulting from increased sparsity, we design a two-stage post-training strategy to enhance model performance. Experiments on the LLaMA3 model demonstrate the potential effectiveness of this approach for future developments of instructed MoE models. The source codes and models are available at: urlhttps://github.com/OpenSparseLLMs/LLaMA-MoE-v2.