mlsanityData to 7 Oct 2026

HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference

arXiv 2411.01433 ↗First posted 3 Nov 2024cs.LG cs.DCcited by 53

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

The Mixture-of-Experts (MoE) architecture has demonstrated significant advantages in the era of Large Language Models (LLMs), offering enhanced capabilities with reduced inference costs. However, deploying MoE-based LLMs on memoryconstrained edge devices remains challenging due to their substantial memory requirements. While existing expertoffloading methods alleviate the memory requirements, they often incur significant expert-loading costs or compromise model accuracy. We present HOBBIT, a mixed precision expert offloading system to enable flexible and efficient MoE inference. Our key insight is that dynamically replacing less critical cache-miss experts with low precision versions can substantially reduce expert-loading latency while preserving model accuracy. HOBBIT introduces three innovative techniques that map the natural hierarchy of MoE computation: (1) a token-level dynamic expert loading mechanism, (2) a layer-level adaptive expert prefetching technique, and (3) a sequence-level multidimensional expert caching policy. These innovations fully leverage the benefits of mixedprecision expert inference. By implementing HOBBIT on top of the renowned LLM inference framework Llama.cpp, we evaluate its performance across different edge devices with representative MoE models. The results demonstrate that HOBBIT achieves up to a 9.93x speedup in decoding compared to state-of-the-art MoE offloading systems.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 37 of its 58 references that are in the map. First posted on arXiv 3 Nov 2024.

Builds on

37 of 58 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference19 Aug
  2. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  3. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
  4. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  5. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices28 Aug

Other references · 32

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2024Mixtral of Expertscited by 1,785
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  10. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  11. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  12. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
20 more
  1. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  2. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  3. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  4. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  5. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  6. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  7. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
  8. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  9. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  10. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  11. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 156
  12. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  13. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  14. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
  15. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 101
  16. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 80
  17. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  18. 2022In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Modelscited by 36
  19. 2024Streamlining Redundant Layers to Compress Large Language Modelscited by 32
  20. 2023Adaptive Gating in Mixture-of-Experts based Language Modelscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 25; 2026: 25.

53 papers.

Sort:

By area:

  1. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  2. 2025Advancing Expert Specialization for Better MoEcited by 37
  3. 2025Delta Decompression for MoE-based LLMs Compressioncited by 32
  4. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Designcited by 26
  5. 2024A Survey on Large Language Models for Communication, Network, and Service Management: Application Insights, Challenges, and Future Directionscited by 25
  6. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  7. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoEcited by 17
  8. 2025HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inferencecited by 16
  9. 2024Mixture of Cache-Conditional Experts for Efficient Mobile Device Inferencecited by 14
  10. 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inferencecited by 12
  11. 2025MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Expertscited by 11
  12. 2025FloE: On-the-Fly MoE Inference on Memory-constrained GPUcited by 9
38 more
  1. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanismcited by 9
  2. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  3. 2025MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batchingcited by 6
  4. 2025Dynamic Expert Quantization for Scalable Mixture-of-Experts Inferencecited by 6
  5. 2026Effective MoE-based LLM Compression by Exploiting Heterogeneous Inter-Group Experts Routing Frequency and Information Densitycited by 6
  6. 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inferencecited by 5
  7. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarioscited by 5
  8. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospectscited by 4
  9. 2026PCR: A Prefetch-Enhanced Cache Reuse System for Low-Latency RAG Servingcited by 4
  10. 2025Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compressioncited by 3
  11. 2025Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Modelscited by 3
  12. 2025Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensationcited by 3
  13. 2026ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inferencecited by 3
  14. 2025CoMoE: Collaborative Optimization of Expert Aggregation and Offloading for MoE-based LLMs at Edgecited by 2
  15. 2025Context-Aware Mixture-of-Experts Inference on CXL-Enabled GPU-NDP Systemscited by 2
  16. 2025Remoe: Towards Efficient and Low-Cost MoE Inference in Serverless Computingcited by 2
  17. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCscited by 2
  18. 2026DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edgecited by 2
  19. 2025MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?cited by 1
  20. 2025MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Expertscited by 1
  21. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inferencecited by 1
  22. 2025SliceMoE: Bit-Sliced Expert Caching under Miss-Rate Constraints for Efficient MoE Inferencecited by 1
  23. 2026Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Designcited by 1
  24. 2026Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inferencecited by 1
  25. 2026FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Executioncited by 1
  26. 2026Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Studycited by 1
  27. 2026Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Schedulingcited by 0
  28. 2026HCInfer: An Efficient Inference System via Error Compensation for Resource-Constrained Devicescited by 0
  29. 2026VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloadingcited by 0
  30. 2026Fast MoE Inference via Predictive Prefetching and Expert Replicationcited by 0
  31. 2026CoX-MoE: Coalesced Expert Execution for High-Throughput MoE Inference with AMX-Enabled CPU-GPU Co-Executioncited by 0
  32. 2026SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Modelscited by 0
  33. 2026Reproducible Evaluation of MoE Expert Caching: Replay Semantics, Workload Contamination, and Operating Regimescited by 0
  34. 2026Tied Trit-Planes: Constraining PTQTP to a Uniform Nine-Level Quantizer, with a Persistent Folded Format for Disk-Streamed Mixture-of-Experts Servingcited by 0
  35. 2026APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inferencecited by 0
  36. 2026Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inferencecited by 0
  37. 2026SAEM: Stage-Aware Expert Management for Memory-Efficient MoE Inference in Chain-of-Thought Reasoningcited by 0
  38. 2026PCoMoE: Shifting MoE Inference from Monolithic Expert Selection to Fine-Grained Path Compositioncited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
  2. 2025Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gate
  3. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline
  4. 2026MawForge: Memory-Bounded Expert Materialization for Local Mixture-of-Experts Inference
  5. 2026MoEless: Efficient MoE LLM Serving via Serverless Computing
  6. 2026OLED-MoE: Accelerating MoE-Based dLLM Inference via Inter-Iteration Locality-Aware Expert Offloading
  7. 2026Speculating Experts Accelerates Inference for Mixture-of-Experts
  8. 2026A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference