mlsanityData to 14 Oct 2026

SSD-LLaMA: SSD-Native Inference for Trillion-Parameter MoE at 1+ Token/s on a Consumer PC

arXiv 2609.18110 ↗First posted 16 Sep 2026cs.DCcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Frontier open-weight language models increasingly use Mixture-of-Experts (MoE) architectures to expand model capacity while activating only a small subset of experts per token. Local inference must nevertheless keep the complete expert pool available, which remains far beyond consumer-grade RAM and VRAM capacity even after quantization. SSDs provide practical capacity at this scale, but turning that capacity into executable model memory requires efficient expert delivery, coordinated management of SSD, RAM, and VRAM, and CPU--GPU hybrid execution under bounded bandwidth. We present textitSSD-LLaMA, an SSD-native local MoE inference system that addresses these challenges with an SSD I/O pipeline optimized for expert delivery, a native three-tier storage hierarchy that delivers and retains experts dynamically, and balanced CPU--GPU hybrid execution. textitSSD-LLaMA executes every selected expert without pruning or substitution. Across three frontier MoE model families, textitSSD-LLaMA improves prefill token rate by 1.52×--4.19× and decode token rate by 2.10×--15.58× over the evaluated baselines. We also achieve higher than 1 token/s for running trillion-parameter model with a single RTX 5090 and no more than 32GB RAM.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 17 of its 41 references that are in the map. First posted on arXiv 16 Sep 2026.

Builds on

17 of 41 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design9 Jun
  2. 2026FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices22 Jan
  3. 2025SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitution26 Aug
  4. 2025SSD Offloading for LLM Mixture-of-Experts Weights Considered Harmful in Energy Efficiency9 Aug
  5. 2025BrownoutServe: SLO-Aware Inference Serving under Bursty Workloads for MoE-based LLMs23 Jul
  6. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  7. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching29 Oct
  8. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
  9. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices28 Aug

Other references · 8

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  4. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  5. 2025Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Mergingcited by 28
  6. 2025EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Modelscited by 18
  7. 2026Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacitycited by 17
  8. 2025A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient CPU-GPU Collaborative Inference for MoE-based LLMs on Memory-Limited Systems
  2. 2025MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraints
  3. 2026The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
  4. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  5. 2025DuoServe-MoE: Dual-Phase Expert Prefetch and Caching for LLM Inference QoS Assurance
  6. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs
  7. 2026Beyond Capacity: Scalable MoE LLM Inference via High-Bandwidth Flash with Direct GPU and HBM Paths
  8. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference