# PockEngine: Sparse and Efficient Fine-tuning in a Pocket

arXiv: https://arxiv.org/abs/2310.17752 · HTML: https://mlsanity.com/paper/2310.17752/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 60 references that are in the map.
- First posted on arXiv 26 Oct 2023.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [MCUNet: Tiny Deep Learning on IoT Devices](https://mlsanity.com/paper/2007.10319/) (2020)
- [Surgical Fine-Tuning Improves Adaptation to Distribution Shifts](https://mlsanity.com/paper/2210.11466/) (2022)
- [MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems](https://mlsanity.com/paper/1512.01274/) (2015)
- [On-Device Training Under 256KB Memory](https://mlsanity.com/paper/2206.15472/) (2022)
- [Return of the Devil in the Details: Delving Deep into Convolutional Nets](https://mlsanity.com/paper/1405.3531/) (2014)
- [Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs](https://mlsanity.com/paper/2003.00152/) (2020)
- [Benchmarking TinyML Systems: Challenges and Direction](https://mlsanity.com/paper/2003.04821/) (2020)
- [MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning](https://mlsanity.com/paper/2110.15352/) (2021)
- [K for the Price of 1: Parameter-efficient Multi-task and Transfer Learning](https://mlsanity.com/paper/1810.10703/) (2018)
- [TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning](https://mlsanity.com/paper/2007.11622/) (2020)
- [Ordering Chaos: Memory-Aware Scheduling of Irregularly Wired Neural Networks for Edge Devices](https://mlsanity.com/paper/2003.02369/) (2020)
- [Neural networks on microcontrollers: saving memory at inference via operator reordering](https://mlsanity.com/paper/1910.05110/) (2019)

## Built on by (most cited first, 23 of 23)

- [Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity](https://mlsanity.com/paper/2406.02913/) (2024)
- [PocketLLM: Enabling On-Device Fine-Tuning for Personalized LLMs](https://mlsanity.com/paper/2407.01031/) (2024)
- [PrivateLoRA For Efficient Privacy Preserving LLM](https://mlsanity.com/paper/2311.14030/) (2023)
- [Robust Implementation of Retrieval-Augmented Generation on Edge-based Computing-in-Memory Architectures](https://mlsanity.com/paper/2405.04700/) (2024)
- [Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Order](https://mlsanity.com/paper/2506.04430/) (2025)
- [Embodied Foundation Models at the Edge: A Survey of Deployment Constraints and Mitigation Strategies](https://mlsanity.com/paper/2603.16952/) (2026)
- [Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approach](https://mlsanity.com/paper/2411.05873/) (2024)
- [Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devices](https://mlsanity.com/paper/2412.20004/) (2024)
- [Training on the Fly: On-device Self-supervised Learning aboard Nano-drones within 20 mW](https://mlsanity.com/paper/2408.03168/) (2024)
- [Resource-Efficient Federated Fine-Tuning Large Language Models for Heterogeneous Data](https://mlsanity.com/paper/2503.21213/) (2025)
- [Advancing On-Device Neural Network Training with TinyPropv2: Dynamic, Sparse, and Efficient Backpropagation](https://mlsanity.com/paper/2409.07109/) (2024)
- [MobiZO: Enabling Efficient LLM Fine-Tuning at the Edge via Inference Engines](https://mlsanity.com/paper/2409.15520/) (2024)
- [Towards Energy-Efficiency by Navigating the Trilemma of Energy, Latency, and Accuracy](https://mlsanity.com/paper/2409.04018/) (2024)
- [Skip2-LoRA: A Lightweight On-device DNN Fine-tuning Method for Low-cost Edge Devices](https://mlsanity.com/paper/2410.21073/) (2024)
- [Enabling Weak Client Participation via On-device Knowledge Distillation in Heterogeneous Federated Learning](https://mlsanity.com/paper/2503.11151/) (2025)
- [Onboard Optimization and Learning: A Survey](https://mlsanity.com/paper/2505.08793/) (2025)
- [SALT: A Lightweight Model Adaptation Method for Closed Split Computing Environments](https://mlsanity.com/paper/2506.07355/) (2025)
- [DistilLock: Safeguarding LLMs from Unauthorized Knowledge Distillation on the Edge](https://mlsanity.com/paper/2510.16716/) (2025)
- [Memory-Efficient Structured Backpropagation for On-Device LLM Fine-Tuning](https://mlsanity.com/paper/2602.13069/) (2026)
- [Zero-Order Optimization for LLM Fine-Tuning via Learnable Direction Sampling](https://mlsanity.com/paper/2602.13659/) (2026)
- [Lightweight User-Personalization Method for Closed Split Computing](https://mlsanity.com/paper/2603.14958/) (2026)
- [AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning](https://mlsanity.com/paper/2608.23816/) (2026)
- [Mobility Helps Learning: Unsupervised Model Adaptation for Object Recognition via Movement](https://mlsanity.com/paper/2610.15846/) (2026)

## Abstract (arXiv)

On-device learning and efficient fine-tuning enable continuous and privacy-preserving customization (e.g., locally fine-tuning large language models on personalized data). However, existing training frameworks are designed for cloud servers with powerful accelerators (e.g., GPUs, TPUs) and lack the optimizations for learning on the edge, which faces challenges of resource limitations and edge hardware diversity. We introduce PockEngine: a tiny, sparse and efficient engine to enable fine-tuning on various edge devices. PockEngine supports sparse backpropagation: it prunes the backward graph and sparsely updates the model with measured memory saving and latency reduction while maintaining the model quality. Secondly, PockEngine is compilation first: the entire training graph (including forward, backward and optimization steps) is derived at compile-time, which reduces the runtime overhead and brings opportunities for graph transformations. PockEngine also integrates a rich set of training graph optimizations, thus can further accelerate the training cost, including operator reordering and backend switching. PockEngine supports diverse applications, frontends and hardware backends: it flexibly compiles and tunes models defined in PyTorch/TensorFlow/Jax and deploys binaries to mobile CPU/GPU/DSPs. We evaluated PockEngine on both vision models and large language models. PockEngine achieves up to 15 × speedup over off-the-shelf TensorFlow (Raspberry Pi), 5.6 × memory saving back-propagation (Jetson AGX Orin). Remarkably, PockEngine enables fine-tuning LLaMav2-7B on NVIDIA Jetson AGX Orin at 550 tokens/s, 7.9× faster than the PyTorch.