# SSD-LLaMA: SSD-Native Inference for Trillion-Parameter MoE at 1+ Token/s on a Consumer PC

arXiv: https://arxiv.org/abs/2609.18110 · HTML: https://mlsanity.com/paper/2609.18110/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 17 of its 41 references that are in the map.
- First posted on arXiv 16 Sep 2026.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 17 of 17)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [LLM in a flash: Efficient Large Language Model Inference with Limited Memory](https://mlsanity.com/paper/2312.11514/) (2023)
- [Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention](https://mlsanity.com/paper/2403.19708/) (2024)
- [MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs](https://mlsanity.com/paper/2411.11217/) (2024)
- [EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices](https://mlsanity.com/paper/2308.14352/) (2023)
- [MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache](https://mlsanity.com/paper/2401.14361/) (2024)
- [ProMoE: Fast MoE-based LLM Serving using Proactive Caching](https://mlsanity.com/paper/2410.22134/) (2024)
- [Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging](https://mlsanity.com/paper/2506.23266/) (2025)
- [EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models](https://mlsanity.com/paper/2508.01625/) (2025)
- [Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity](https://mlsanity.com/paper/2604.24827/) (2026)
- [A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMs](https://mlsanity.com/paper/2502.09921/) (2025)
- [BrownoutServe: SLO-Aware Inference Serving under Bursty Workloads for MoE-based LLMs](https://mlsanity.com/paper/2507.17133/) (2025)
- [SSD Offloading for LLM Mixture-of-Experts Weights Considered Harmful in Energy Efficiency](https://mlsanity.com/paper/2508.06978/) (2025)
- [SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitution](https://mlsanity.com/paper/2508.18983/) (2025)
- [Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design](https://mlsanity.com/paper/2606.10493/) (2026)
- [FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices](https://mlsanity.com/paper/2601.17063/) (2026)

## Abstract (arXiv)

Frontier open-weight language models increasingly use Mixture-of-Experts (MoE) architectures to expand model capacity while activating only a small subset of experts per token. Local inference must nevertheless keep the complete expert pool available, which remains far beyond consumer-grade RAM and VRAM capacity even after quantization. SSDs provide practical capacity at this scale, but turning that capacity into executable model memory requires efficient expert delivery, coordinated management of SSD, RAM, and VRAM, and CPU--GPU hybrid execution under bounded bandwidth. We present textitSSD-LLaMA, an SSD-native local MoE inference system that addresses these challenges with an SSD I/O pipeline optimized for expert delivery, a native three-tier storage hierarchy that delivers and retains experts dynamically, and balanced CPU--GPU hybrid execution. textitSSD-LLaMA executes every selected expert without pruning or substitution. Across three frontier MoE model families, textitSSD-LLaMA improves prefill token rate by 1.52×--4.19× and decode token rate by 2.10×--15.58× over the evaluated baselines. We also achieve higher than 1 token/s for running trillion-parameter model with a single RTX 5090 and no more than 32GB RAM.