# Enabling On-Device CNN Training by Self-Supervised Instance Filtering and Error Map Pruning

arXiv: https://arxiv.org/abs/2007.03213 · HTML: https://mlsanity.com/paper/2007.03213/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jul 2020.

## Built on by (most cited first, 24 of 24)

- [AutoReP: Automatic ReLU Replacement for Fast Private Network Inference](https://mlsanity.com/paper/2308.10134/) (2023)
- [Machine Learning for Microcontroller-Class Hardware: A Review](https://mlsanity.com/paper/2205.14550/) (2022)
- [LinGCN: Structural Linearized Graph Convolutional Network for Homomorphically Encrypted Inference](https://mlsanity.com/paper/2309.14331/) (2023)
- [Repeated Random Sampling for Minimizing the Time-to-Accuracy of Learning](https://mlsanity.com/paper/2305.18424/) (2023)
- [Green Edge AI: A Contemporary Survey](https://mlsanity.com/paper/2312.00333/) (2023)
- [Enabling On-Device Self-Supervised Contrastive Learning With Selective Data Contrast](https://mlsanity.com/paper/2106.03796/) (2021)
- [Federated Contrastive Learning for Dermatological Disease Diagnosis via On-device Learning](https://mlsanity.com/paper/2202.07470/) (2022)
- [Unlocking Memorization in Large Language Models with Dynamic Soft Prompting](https://mlsanity.com/paper/2409.13853/) (2024)
- [An Accelerated Doubly Stochastic Gradient Method with Faster Explicit Model Identification](https://mlsanity.com/paper/2208.06058/) (2022)
- [RRNet: Towards ReLU-Reduced Neural Network for Two-party Computation Based Private Inference](https://mlsanity.com/paper/2302.02292/) (2023)
- [SmartFRZ: An Efficient Training Framework using Attention-Based Layer Freezing](https://mlsanity.com/paper/2401.16720/) (2024)
- [Exploration of Quantum Neural Architecture by Mixing Quantum Neuron Designs](https://mlsanity.com/paper/2109.03806/) (2021)
- [Watermark under Fire: A Robustness Evaluation of LLM Watermarking](https://mlsanity.com/paper/2411.13425/) (2024)
- [Enabling Resource-efficient AIoT System with Cross-level Optimization: A survey](https://mlsanity.com/paper/2309.15467/) (2023)
- [Self-supervised On-device Federated Learning from Unlabeled Streams](https://mlsanity.com/paper/2212.01006/) (2022)
- [QuMoS: A Framework for Preserving Security of Quantum Machine Learning Model](https://mlsanity.com/paper/2304.11511/) (2023)
- [Federated Self-Supervised Contrastive Learning and Masked Autoencoder for Dermatological Disease Diagnosis](https://mlsanity.com/paper/2208.11278/) (2022)
- [Efficient N:M Sparse DNN Training Using Algorithm, Architecture, and Dataflow Co-Design](https://mlsanity.com/paper/2309.13015/) (2023)
- [Achieving Fairness in Dermatological Disease Diagnosis through Automatic Weight Adjusting Federated Learning and Personalization](https://mlsanity.com/paper/2208.11187/) (2022)
- [All-in-One Tuning and Structural Pruning for Domain-Specific LLMs](https://mlsanity.com/paper/2412.14426/) (2024)
- [Edge-InversionNet: Enabling Efficient Inference of InversionNet on Edge Devices](https://mlsanity.com/paper/2310.09667/) (2023)
- [KAKURENBO: Adaptively Hiding Samples in Deep Neural Network Training](https://mlsanity.com/paper/2310.10102/) (2023)
- [A Self-guided Multimodal Approach to Enhancing Graph Representation Learning for Alzheimer's Diseases](https://mlsanity.com/paper/2412.06212/) (2024)
- [Enabling Weakly-Supervised Temporal Action Localization from On-Device Learning of the Video Stream](https://mlsanity.com/paper/2208.12673/) (2022)

## Abstract (arXiv)

This work aims to enable on-device training of convolutional neural networks (CNNs) by reducing the computation cost at training time. CNN models are usually trained on high-performance computers and only the trained models are deployed to edge devices. But the statically trained model cannot adapt dynamically in a real environment and may result in low accuracy for new inputs. On-device training by learning from the real-world data after deployment can greatly improve accuracy. However, the high computation cost makes training prohibitive for resource-constrained devices. To tackle this problem, we explore the computational redundancies in training and reduce the computation cost by two complementary approaches: self-supervised early instance filtering on data level and error map pruning on the algorithm level. The early instance filter selects important instances from the input stream to train the network and drops trivial ones. The error map pruning further prunes out insignificant computations when training with the selected instances. Extensive experiments show that the computation cost is substantially reduced without any or with marginal accuracy loss. For example, when training ResNet-110 on CIFAR-10, we achieve 68% computation saving while preserving full accuracy and 75% computation saving with a marginal accuracy loss of 1.3%. Aggressive computation saving of 96% is achieved with less than 0.1% accuracy loss when quantization is integrated into the proposed approaches. Besides, when training LeNet on MNIST, we save 79% computation while boosting accuracy by 0.2%.