# Skip2-LoRA: A Lightweight On-device DNN Fine-tuning Method for Low-cost Edge Devices

arXiv: https://arxiv.org/abs/2410.21073 · HTML: https://mlsanity.com/paper/2410.21073/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 14 references that are in the map.
- First posted on arXiv 28 Oct 2024.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [TinyOL: TinyML with Online-Learning on Microcontrollers](https://mlsanity.com/paper/2103.08295/) (2021)
- [PockEngine: Sparse and Efficient Fine-tuning in a Pocket](https://mlsanity.com/paper/2310.17752/) (2023)
- [A Neural Network-Based On-device Learning Anomaly Detector for Edge Devices](https://mlsanity.com/paper/1907.10147/) (2019)
- [Reduced Precision Floating-Point Optimization for Deep Neural Network On-Device Learning on MicroControllers](https://mlsanity.com/paper/2305.19167/) (2023)
- [Addressing Gap between Training Data and Deployed Environment by On-Device Learning](https://mlsanity.com/paper/2203.01077/) (2022)

## Built on by (most cited first, 1 of 1)

- [InstantFT: An FPGA-Based Runtime Subsecond Fine-tuning of CNN Models](https://mlsanity.com/paper/2506.06505/) (2025)

## Abstract (arXiv)

This paper proposes Skip2-LoRA as a lightweight fine-tuning method for deep neural networks to address the gap between pre-trained and deployed models. In our approach, trainable LoRA (low-rank adaptation) adapters are inserted between the last layer and every other layer to enhance the network expressive power while keeping the backward computation cost low. This architecture is well-suited to cache intermediate computation results of the forward pass and then can skip the forward computation of seen samples as training epochs progress. We implemented the combination of the proposed architecture and cache, denoted as Skip2-LoRA, and tested it on a 15 single board computer. Our results show that Skip2-LoRA reduces the fine-tuning time by 90.0% on average compared to the counterpart that has the same number of trainable parameters while preserving the accuracy, while taking only a few seconds on the microcontroller board.