mlsanityData to 7 Oct 2026

GRIN: GRadient-INformed MoE

arXiv 2409.12136 ↗First posted 18 Sep 2024cs.CL cs.AI cs.LGcited by 16

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-Experts (MoE) models scale more effectively than dense models due to sparse computation through expert routing, selectively activating only a small subset of expert modules. However, sparse computation challenges traditional training practices, as discrete expert routing hinders standard backpropagation and thus gradient-based optimization, which are the cornerstone of deep learning. To better pursue the scaling power of MoE, we introduce GRIN (GRadient-INformed MoE training), which incorporates sparse gradient estimation for expert routing and configures model parallelism to avoid token dropping. Applying GRIN to autoregressive language modeling, we develop a top-2 16×3.8B MoE model. Our model, with only 6.6B activated parameters, outperforms a 7B dense model and matches the performance of a 14B dense model trained on the same data. Extensive evaluations across diverse tasks demonstrate the potential of GRIN to significantly enhance MoE efficacy, achieving 79.4 on MMLU, 83.7 on HellaSwag, 74.4 on HumanEval, and 58.9 on MATH.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 35 of its 55 references that are in the map. First posted on arXiv 18 Sep 2024.

Builds on

35 of 55 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models3 Jun
  2. 2024Mixtral of Experts8 Jan
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  4. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models22 Sep
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2021Program Synthesis with Large Language Modelscited by 3,509
  10. 2023Mistral 7Bcited by 3,065
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
17 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  2. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  6. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  7. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  8. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  9. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  10. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  11. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  12. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
  13. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  14. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 99
  15. 2024Phi-3 Safety Post-Training: Aligning Language Models with a "Break-Fix" Cyclecited by 32
  16. 2023Bridging Discrete and Backpropagation: Straight-Through and Beyondcited by 31
  17. 2023Sparse Backpropagation for MoE Trainingcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 5.

16 papers.

Sort:

By area:

  1. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 53
  2. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  3. 2026Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion Scalecited by 25
  4. 2025Instruction-Following Pruning for Large Language Modelscited by 14
  5. 2025Scaling Laws for Upcycling Mixture-of-Experts Language Modelscited by 12
  6. 2025Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUscited by 12
  7. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  8. 2025BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsitycited by 6
  9. 2025Continual Pre-training of MoEs: How robust is your router?cited by 5
  10. 2025Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Expertscited by 3
  11. 2025FLIP Reasoning Challengecited by 1
  12. 2025An Introduction to Discrete Variational Autoencoderscited by 1
4 more
  1. 2026PR2: Predictive Routing Replay for MoE-Based LLM Reinforcement Learningcited by 1
  2. 2026Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Schedulingcited by 0
  3. 2026UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Modelscited by 0
  4. 2026Chimaera: A Mixture-of-Graph-Experts Architecture for Cross-Task and Cross-Dataset Graph Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Advancing Expert Specialization for Better MoE
  2. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  3. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  4. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models
  5. 2025Sigma-MoE-Tiny Technical Report
  6. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
  7. 2025Mixture of Experts in Large Language Models
  8. 2024Scaling Laws for Fine-Grained Mixture of Experts