# Mandheling: Mixed-Precision On-Device DNN Training with DSP Offloading

arXiv: https://arxiv.org/abs/2206.07509 · HTML: https://mlsanity.com/paper/2206.07509/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jun 2022.

## Built on by (most cited first, 29 of 29)

- [Fast On-device LLM Inference with NPUs](https://mlsanity.com/paper/2407.05858/) (2024)
- [Tiny Machine Learning: Progress and Futures](https://mlsanity.com/paper/2403.19076/) (2024)
- [On-device Training: A First Overview on Existing Systems](https://mlsanity.com/paper/2212.00824/) (2022)
- [TinyTrain: Resource-Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edge](https://mlsanity.com/paper/2307.09988/) (2023)
- [Federated Few-Shot Learning for Mobile NLP](https://mlsanity.com/paper/2212.05974/) (2022)
- [Asteroid: Resource-Efficient Hybrid Pipeline Parallelism for Collaborative DNN Training on Heterogeneous Edge Devices](https://mlsanity.com/paper/2408.08015/) (2024)
- [ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection](https://mlsanity.com/paper/2312.14227/) (2023)
- [Lifelong Intelligence Beyond the Edge using Hyperdimensional Computing](https://mlsanity.com/paper/2403.04759/) (2024)
- [On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollers](https://mlsanity.com/paper/2407.10734/) (2024)
- [From LLMs to Edge: Parameter-Efficient Fine-Tuning on Edge Devices](https://mlsanity.com/paper/2507.23536/) (2025)
- [Low-Energy On-Device Personalization for MCUs](https://mlsanity.com/paper/2403.08040/) (2024)
- [Edge Graph Intelligence: Reciprocally Empowering Edge Networks with Graph Intelligence](https://mlsanity.com/paper/2407.15320/) (2024)
- [FedEL: Federated Elastic Learning for Heterogeneous Devices](https://mlsanity.com/paper/2509.16902/) (2025)
- [Tin-Tin: Towards Tiny Learning on Tiny Devices with Integer-based Neural Network Training](https://mlsanity.com/paper/2504.09405/) (2025)
- [Recursive Offloading for LLM Serving in Multi-tier Networks](https://mlsanity.com/paper/2505.16502/) (2025)
- [A Two-Stage Data Selection Framework for Data-Efficient Model Training on Edge Devices](https://mlsanity.com/paper/2505.16563/) (2025)
- [Breaking On-device Training Memory Wall: A Systematic Survey](https://mlsanity.com/paper/2306.10388/) (2023)
- [Quantized Distillation: Optimizing Driver Activity Recognition Models for Resource-Constrained Environments](https://mlsanity.com/paper/2311.05970/) (2023)
- [Resource-Efficient Personal Large Language Models Fine-Tuning with Collaborative Edge Computing](https://mlsanity.com/paper/2408.10746/) (2024)
- [AdaShadow: Responsive Test-time Model Adaptation in Non-stationary Mobile Environments](https://mlsanity.com/paper/2410.08256/) (2024)
- [LeanTTA: A Backpropagation-Free and Stateless Approach to Quantized Test-Time Adaptation on Edge Devices](https://mlsanity.com/paper/2503.15889/) (2025)
- [Optimizing Storage Overhead of User Behavior Log for ML-embedded Mobile Apps](https://mlsanity.com/paper/2510.13405/) (2025)
- [Online Resource Allocation for Edge Intelligence with Colocated Model Retraining and Inference](https://mlsanity.com/paper/2405.16029/) (2024)
- [THOR: A Generic Energy Estimation Approach for On-Device Training](https://mlsanity.com/paper/2501.16397/) (2025)
- [Moss: Proxy Model-based Full-Weight Aggregation in Federated Learning with Heterogeneous Models](https://mlsanity.com/paper/2503.10218/) (2025)
- [Comprehensive Design Space Exploration for Tensorized Neural Network Hardware Accelerators](https://mlsanity.com/paper/2511.17971/) (2025)
- [Optimizing Feature Extraction for On-device Model Inference with User Behavior Sequences](https://mlsanity.com/paper/2603.21508/) (2026)
- [Beyond Benchmarks: Continuous Edge Inference for Fine-Grained Roadside Perception](https://mlsanity.com/paper/2606.17241/) (2026)
- [FBLayout: Optimizing Memory Layout for Efficient LLM Finetuning on Mobile GPUs](https://mlsanity.com/paper/2607.21624/) (2026)

## Abstract (arXiv)

This paper proposes Mandheling, the first system that enables highly resource-efficient on-device training by orchestrating the mixed-precision training with on-chip Digital Signal Processing (DSP) offloading. Mandheling fully explores the advantages of DSP in integer-based numerical calculation by four novel techniques: (1) a CPU-DSP co-scheduling scheme to mitigate the overhead from DSP-unfriendly operators; (2) a self-adaptive rescaling algorithm to reduce the overhead of dynamic rescaling in backward propagation; (3) a batch-splitting algorithm to improve the DSP cache efficiency; (4) a DSP-compute subgraph reusing mechanism to eliminate the preparation overhead on DSP. We have fully implemented Mandheling and demonstrate its effectiveness through extensive experiments. The results show that, compared to the state-of-the-art DNN engines from TFLite and MNN, Mandheling reduces the per-batch training time by 5.5× and the energy consumption by 8.9× on average. In end-to-end training tasks, Mandheling reduces up to 10.7× convergence time and 13.1× energy consumption, with only 1.9%-2.7% accuracy loss compared to the FP32 precision setting.