# Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flash

arXiv: https://arxiv.org/abs/2504.08378 · HTML: https://mlsanity.com/paper/2504.08378/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 29 references that are in the map.
- First posted on arXiv 11 Apr 2025.

## Builds on (most cited first, 22 of 22)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023)
- [Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention](https://mlsanity.com/paper/2502.11089/) (2025)
- [Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time](https://mlsanity.com/paper/2310.17157/) (2023)
- [InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management](https://mlsanity.com/paper/2406.19707/) (2024)
- [PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU](https://mlsanity.com/paper/2312.12456/) (2023)
- [LLM in a flash: Efficient Large Language Model Inference with Limited Memory](https://mlsanity.com/paper/2312.11514/) (2023)
- [SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs](https://mlsanity.com/paper/2410.13276/) (2024)
- [PowerInfer-2: Fast Large Language Model Inference on a Smartphone](https://mlsanity.com/paper/2406.06282/) (2024)
- [BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation](https://mlsanity.com/paper/2402.10631/) (2024)
- [T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge](https://mlsanity.com/paper/2407.00088/) (2024)
- [Training-Free Activation Sparsity in Large Language Models](https://mlsanity.com/paper/2408.14690/) (2024)
- [Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters](https://mlsanity.com/paper/2406.05955/) (2024)
- [DB-LLM: Accurate Dual-Binarization for Efficient LLMs](https://mlsanity.com/paper/2402.11960/) (2024)
- [Q-Sparse: All Large Language Models can be Fully Sparsely-Activated](https://mlsanity.com/paper/2407.10969/) (2024)
- [Bitnet.cpp: Efficient Edge Inference for Ternary LLMs](https://mlsanity.com/paper/2502.11880/) (2025)
- [Poster: Self-Supervised Quantization-Aware Knowledge Distillation](https://mlsanity.com/paper/2309.13220/) (2023)
- [CFSP: An Efficient Structured Pruning Framework for LLMs with Coarse-to-Fine Activation Information](https://mlsanity.com/paper/2409.13199/) (2024)
- [Boosting DNN Cold Inference on Edge Devices](https://mlsanity.com/paper/2206.07446/) (2022)
- [Neuralink: Fast LLM Inference on Smartphones with Neuron Co-Activation Linking](https://mlsanity.com/paper/2410.19274/) (2024)
- [Regularization-based Framework for Quantization-, Fault- and Variability-Aware Training](https://mlsanity.com/paper/2503.01297/) (2025)

## Built on by (most cited first, 4 of 4)

- [DynaKV: Enabling Accurate and Efficient Long-Sequence LLM Decoding on Smartphones](https://mlsanity.com/paper/2511.07427/) (2025)
- [Chain-based Distillation for Effective Initialization of Variable-Sized Small Language Models](https://mlsanity.com/paper/2605.07783/) (2026)
- [JustFit: Just-in-Time State Management for Local LLM Serving](https://mlsanity.com/paper/2609.17475/) (2026)
- [Activation Sparsity with Weight Approximation for Faster LLM Decoding on Offloaded Weights](https://mlsanity.com/paper/2610.02598/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly being deployed on mobile devices, but the limited DRAM capacity constrains the deployable model size. This paper introduces ActiveFlow, the first LLM inference framework that can achieve adaptive DRAM usage for modern LLMs (not ReLU-based), enabling the scaling up of deployable model sizes. The framework is based on the novel concept of active weight DRAM-flash swapping and incorporates three novel techniques: (1) Cross-layer active weights preloading. It uses the activations from the current layer to predict the active weights of several subsequent layers, enabling computation and data loading to overlap, as well as facilitating large I/O transfers. (2) Sparsity-aware self-distillation. It adjusts the active weights to align with the dense-model output distribution, compensating for approximations introduced by contextual sparsity. (3) Active weight DRAM-flash swapping pipeline. It orchestrates the DRAM space allocation among the hot weight cache, preloaded active weights, and computation-involved weights based on available memory. Results show ActiveFlow achieves the performance-cost Pareto frontier compared to existing efficiency optimization methods.