Techniques for Peak Memory Reduction for LoRA Fine-tuning of LLMs on Edge Devices
Not on the map. Why?
Abstract (arXiv)
Fine-tuning of Large Language Models (LLMs) using Low-Rank Adaptation (LoRA) on an end-user's data offers personalized experiences while keeping data private, but faces severe memory constraints on consumer hardware. Peak memory during fine-tuning often exceeds device limits, especially for models with billions of parameters and long-context training data. This paper introduces a suite of complementary techniques to reduce memory footprint without sacrificing model quality: (1) base model quantization with on-the-fly dequantization, (2) memory-efficient checkpointing combining selective activation caching and disk offloading, (3) softmax approximation using semantically relevant token subsets, and (4) logits masking. Experiments on Llama-3.2 3B and Qwen-2.5 3B demonstrate up to 26× and 28× reduction in peak memory, enabling fine-tuning on resource-constrained devices.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 29 references that are in the map. First posted on arXiv 17 Jun 2026.
Builds on
23 of 29 references.
References · 23
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
- 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2023Mistral 7Bcited by 3,164
- 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
- 2016Training Deep Nets with Sublinear Memory Costcited by 848
- 2023Fine-Tuning Language Models with Just Forward Passescited by 345
11 more
- 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
- 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
- 2016Efficient softmax approximation for GPUscited by 67
- 2024Instruction Fine-Tuning: Does Prompt Loss Matter?cited by 26
- 2024PocketLLM: Enabling On-Device Fine-Tuning for Personalized LLMscited by 18
- 2020Memory Optimization for Deep Networkscited by 15
- 2025MobiLLM: Enabling LLM Fine-Tuning on the Mobile Device via Server Assisted Side Tuningcited by 10
- 2021Efficient Softmax Approximation for Deep Neural Networks with Attention Mechanismcited by 9
- 2022A New Frontier of AI: On-Device AI Training and Personalizationcited by 5
- 2025From LLMs to Edge: Parameter-Efficient Fine-Tuning on Edge Devicescited by 4
- 2025Adacc: An Adaptive Framework Unifying Compression and Activation Recomputation for LLM Trainingcited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024EDGE-LLM: Enabling Efficient Large Language Model Adaptation on Edge Devices via Layerwise Unified Compression and Adaptive Layer Tuning and Voting
- 2026Parameter Efficiency Is Not Memory Efficiency: Rethinking Fine-Tuning for On-Device LLM Adaptation
- 2024LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report
- 2025LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
- 2026Advancing Model Refinement: Muon-Optimized Distillation and Quantization for LLM Deployment
- 2023INT2.1: Towards Fine-Tunable Quantized Large Language Models with Error Correction through Low-Rank Adaptation
- 2025EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices
- 2023ModuLoRA: Finetuning 2-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers