mlsanityData to 14 Oct 2026

FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution

arXiv 2608.16157 ↗First posted 17 Aug 2026cs.DCcited by 3

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Frontier open-weight models are increasingly available, but serving them still largely assumes datacenter infrastructure. We present FreeToken, an edge-native MoE serving system that treats a personal machine not as a small GPU, but as a unified, elastic inference platform. FreeToken co-designs the full serving stack, including model layout and loading, expert residency, CPU--GPU execution, agentic state reuse, and runtime memory management, around two realities of local AI: agent workloads continuously change their execution pattern, and edge hardware exposes heterogeneous resources whose balance differs from machine to machine. Rather than committing to a fixed offloading strategy, FreeToken continuously maps computation and model state onto the resources actually available. FreeToken supports more than 20 MoE models and real coding and tool-using agents across hardware ranging from an 8GB laptop GPU to a single workstation GPU. More importantly, it changes what these machines can practically serve, from a 35B model on a laptop to a 284B model on a gaming desktop and the 753B GLM-5.2 on a single workstation GPU. FreeToken turns open weights into deployable local software, making the machines users already own a practical platform for frontier-scale intelligence. We release the system at flashml.ai.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 20 of its 39 references that are in the map. First posted on arXiv 17 Aug 2026.

Builds on

20 of 39 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitution26 Aug
  2. 2025HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference8 Apr
  3. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline9 Feb
  4. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading7 Feb
  5. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  6. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
  7. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching29 Oct
  8. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  9. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
  10. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  11. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug

Other references · 9

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  4. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  5. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  6. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  7. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  8. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  9. 2026TraceLab: Characterizing Coding Agent Workloads for LLM Servingcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026WiSDoM: Wireless Sparse Decision Transformer with Mixture-of-Experts for Multi-Task Mobile Network Optimizationcited by 0
  2. 2026What Does a Harness Repair? A Preregistered Study of Visibility, Baseline Adequacy and Evaluation Defectscited by 0
  3. 2026DuoDiscover: Scaling Autonomous Discovery Beyond the Evaluation Bottleneckcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From PyTorch to the NPU: LLM-Agent-Driven Model Conversion Across Heterogeneous Inference Runtimes
  2. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs
  3. 2025MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designs
  4. 2025Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Scheduling
  5. 2026Moebius: Serving Mixture-of-Expert Models with Seamless Runtime Parallelism Switch
  6. 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems
  7. 2026Separating Intelligence from Inference: A Standard for Edge-Native AI Computing ↗
  8. 2026SSD-LLaMA: SSD-Native Inference for Trillion-Parameter MoE at 1+ Token/s on a Consumer PC