# Techniques for Peak Memory Reduction for LoRA Fine-tuning of LLMs on Edge Devices

arXiv: https://arxiv.org/abs/2606.19528 · HTML: https://mlsanity.com/paper/2606.19528/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 29 references that are in the map.
- First posted on arXiv 17 Jun 2026.

## Builds on (most cited first, 23 of 23)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [Fine-Tuning Language Models with Just Forward Passes](https://mlsanity.com/paper/2305.17333/) (2023)
- [ZeRO-Offload: Democratizing Billion-Scale Model Training](https://mlsanity.com/paper/2101.06840/) (2021)
- [MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases](https://mlsanity.com/paper/2402.14905/) (2024)
- [Efficient softmax approximation for GPUs](https://mlsanity.com/paper/1609.04309/) (2016)
- [Instruction Fine-Tuning: Does Prompt Loss Matter?](https://mlsanity.com/paper/2401.13586/) (2024)
- [PocketLLM: Enabling On-Device Fine-Tuning for Personalized LLMs](https://mlsanity.com/paper/2407.01031/) (2024)
- [Memory Optimization for Deep Networks](https://mlsanity.com/paper/2010.14501/) (2020)
- [MobiLLM: Enabling LLM Fine-Tuning on the Mobile Device via Server Assisted Side Tuning](https://mlsanity.com/paper/2502.20421/) (2025)
- [Efficient Softmax Approximation for Deep Neural Networks with Attention Mechanism](https://mlsanity.com/paper/2111.10770/) (2021)
- [A New Frontier of AI: On-Device AI Training and Personalization](https://mlsanity.com/paper/2206.04688/) (2022)
- [From LLMs to Edge: Parameter-Efficient Fine-Tuning on Edge Devices](https://mlsanity.com/paper/2507.23536/) (2025)
- [Adacc: An Adaptive Framework Unifying Compression and Activation Recomputation for LLM Training](https://mlsanity.com/paper/2508.00806/) (2025)

## Abstract (arXiv)

Fine-tuning of Large Language Models (LLMs) using Low-Rank Adaptation (LoRA) on an end-user's data offers personalized experiences while keeping data private, but faces severe memory constraints on consumer hardware. Peak memory during fine-tuning often exceeds device limits, especially for models with billions of parameters and long-context training data. This paper introduces a suite of complementary techniques to reduce memory footprint without sacrificing model quality: (1) base model quantization with on-the-fly dequantization, (2) memory-efficient checkpointing combining selective activation caching and disk offloading, (3) softmax approximation using semantically relevant token subsets, and (4) logits masking. Experiments on Llama-3.2 3B and Qwen-2.5 3B demonstrate up to 26× and 28× reduction in peak memory, enabling fine-tuning on resource-constrained devices.