mlsanityData to 14 Oct 2026

Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design

arXiv 2410.19123 ↗First posted 24 Oct 2024cs.CL cs.LGcited by 19

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

The proliferation of large language models (LLMs) has led to the adoption of Mixture-of-Experts (MoE) architectures that dynamically leverage specialized subnetworks for improved efficiency and performance. Despite their benefits, MoE models face significant challenges during inference, including inefficient memory management and suboptimal batching, due to misaligned design choices between the model architecture and the system policies. Furthermore, the conventional approach of training MoEs from scratch is increasingly prohibitive in terms of cost. In this paper, we propose a novel framework Read-ME that transforms pre-trained dense LLMs into smaller MoE models (in contrast to "upcycling" generalist MoEs), avoiding the high costs of ground-up training. Our approach employs activation sparsity to extract experts. To compose experts, we examine the widely-adopted layer-wise router design and show its redundancy, and thus we introduce the pre-gating router decoupled from the MoE backbone that facilitates system-friendly pre-computing and lookahead scheduling, enhancing expert-aware batching and caching. Our codesign therefore addresses critical gaps on both the algorithmic and system fronts, establishing a scalable and efficient alternative for LLM inference in resource-constrained settings. Read-ME outperforms other popular open-source dense models of similar scales, achieving improvements of up to 10.1% on MMLU, and improving mean end-to-end latency up to 6.1%. Codes are available at: https://github.com/VITA-Group/READ-ME.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 41 of its 64 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

41 of 64 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  2. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug

Other references · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2023Mistral 7Bcited by 3,164
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  12. 2024Mixtral of Expertscited by 1,823
27 more
  1. 2020GLU Variants Improve Transformercited by 1,804
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2019Root Mean Square Layer Normalizationcited by 1,480
  4. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  5. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  6. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  7. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  8. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  9. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  10. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  11. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  12. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  13. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  14. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  15. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  16. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 152
  17. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 126
  18. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 122
  19. 2019On the Use of ArXiv as a Datasetcited by 103
  20. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 98
  21. 2023The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Mattercited by 42
  22. 2023Emergent Modularity in Pre-trained Transformerscited by 37
  23. 2023Robust Mixture-of-Expert Training for Convolutional Neural Networkscited by 30
  24. 2023Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Expertscited by 23
  25. 2024Learn To be Efficient: Build Structured Sparsity in Large Language Modelscited by 20
  26. 2023Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Modelscited by 19
  27. 2024HiRE: High Recall Approximate Top-k Estimation for Efficient LLM Inferencecited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 9.

19 papers.

Sort:

By area:

  1. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  2. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  3. 2025Life-Cycle Routing Vulnerabilities of LLM Routercited by 11
  4. 2025Too Helpful, Too Harmless, Too Honest or Just Right?cited by 9
  5. 2025Analytical FFN-to-MoE Restructuring via Activation Pattern Analysiscited by 8
  6. 2025SlimCaching: Edge Caching of Mixture-of-Experts for Distributed Inferencecited by 7
  7. 2026Synergistic Intra- and Cross-Layer Regularization Losses for MoE Expert Specializationcited by 5
  8. 2026Grouter: Decoupling Routing from Representation for Accelerated MoE Trainingcited by 5
  9. 2025Toward Cost-Efficient Serving of Mixture-of-Experts with Asynchronycited by 3
  10. 2025On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attentioncited by 3
  11. 2025Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensationcited by 3
  12. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCscited by 2
7 more
  1. 2025HybridServe: Efficient Serving of Large AI Models with Confidence-Based Cascade Routingcited by 1
  2. 2026When the Model Said 'No Comment', We Knew Helpfulness Was Dead, Honesty Was Alive, and Safety Was Terrifiedcited by 1
  3. 2026Teacher-Guided Routing for Sparse Vision Mixture-of-Expertscited by 0
  4. 2026EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decodingcited by 0
  5. 2026Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEscited by 0
  6. 2026Beyond the Previous Layer: Residual Predictive Structure in Sparse MoE Routingcited by 0
  7. 2026One Model to Loop Them All: Elastic Latent Recurrence for Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing
  2. 2024Performance Characterization of Expert Router for Scalable LLM Inference
  3. 2025DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs
  4. 2026ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMs
  5. 2025Multilingual Routing in Mixture-of-Experts
  6. 2024Layerwise Recurrent Router for Mixture-of-Experts
  7. 2026A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference
  8. 2026A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs