mlsanityData to 14 Oct 2026

Mixture of Lookup Experts

arXiv 2503.15798 ↗First posted 20 Mar 2025cs.LG cs.CLcited by 12

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) activates only a subset of experts during inference, allowing the model to maintain low inference FLOPs and latency even as the parameter count scales up. However, since MoE dynamically selects the experts, all the experts need to be loaded into VRAM. Their large parameter size still limits deployment, and offloading, which load experts into VRAM only when needed, significantly increase inference latency. To address this, we propose Mixture of Lookup Experts (MoLE), a new MoE architecture that is efficient in both communication and VRAM usage. In MoLE, the experts are Feed-Forward Networks (FFNs) during training, taking the output of the embedding layer as input. Before inference, these experts can be re-parameterized as lookup tables (LUTs) that retrieves expert outputs based on input ids, and offloaded to storage devices. Therefore, we do not need to perform expert computations during inference. Instead, we directly retrieve the expert's computation results based on input ids and load them into VRAM, and thus the resulting communication overhead is negligible. Experiments show that, with the same FLOPs and VRAM usage, MoLE achieves inference speeds comparable to dense models and significantly faster than MoE with experts offloading, while maintaining performance on par with MoE.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 22 of its 33 references that are in the map. First posted on arXiv 20 Mar 2025.

Builds on

22 of 33 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts9 Oct
  2. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  3. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug

Other references · 19

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  9. 2024Mixtral of Expertscited by 1,823
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  11. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  12. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
7 more
  1. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  2. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 372
  3. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  4. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
  5. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  6. 2015Distributed Gaussian Processescited by 84
  7. 2015Generative Image Modeling Using Spatial LSTMscited by 35

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 10.

12 papers.

  1. 2026MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformerscited by 5
  2. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarioscited by 5
  3. 2026Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memorycited by 5
  4. 2025Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compressioncited by 3
  5. 2026JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulationcited by 3
  6. 2026Scaling Continual Learning to 300+ Tasks with Bi-Level Routing Mixture-of-Expertscited by 3
  7. 2025Mixture of Lookup Key-Value Expertscited by 1
  8. 2026Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcyclingcited by 0
  9. 2026TIDE: Every Layer Knows the Token Beneath the Contextcited by 0
  10. 2026Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligencecited by 0
  11. 2026Budgeting Bytes: A Windowed Storage Roofline and Dual-Budget Architecture Ablations for Storage-Bound LLM Decodingcited by 0
  12. 2026ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  2. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  3. 2026Speculating Experts Accelerates Inference for Mixture-of-Experts
  4. 2025Janus: Disaggregating Attention and Experts for Scalable MoE Inference
  5. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  6. 2025BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inference
  7. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference
  8. 2025Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts