# Enabling On-Device Self-Supervised Contrastive Learning With Selective Data Contrast

arXiv: https://arxiv.org/abs/2106.03796 · HTML: https://mlsanity.com/paper/2106.03796/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 7 Jun 2021.

## Builds on (most cited first, 19 of 19)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [On Tiny Episodic Memories in Continual Learning](https://mlsanity.com/paper/1902.10486/) (2019)
- [Coresets via Bilevel Optimization for Continual Learning and Streaming](https://mlsanity.com/paper/2006.03875/) (2020)
- [Memory Efficient Experience Replay for Streaming Learning](https://mlsanity.com/paper/1809.05922/) (2018)
- [Accelerating Deep Learning by Focusing on the Biggest Losers](https://mlsanity.com/paper/1910.00762/) (2019)
- [Self-supervised learning through the eyes of a child](https://mlsanity.com/paper/2007.16189/) (2020)
- [Hardware/Software Co-Exploration of Neural Architectures](https://mlsanity.com/paper/1907.04650/) (2019)
- [The Curious Robot: Learning Visual Representations via Physical Interactions](https://mlsanity.com/paper/1604.01360/) (2016)
- [Accuracy vs. Efficiency: Achieving Both through FPGA-Implementation Aware Neural Architecture Search](https://mlsanity.com/paper/1901.11211/) (2019)
- [OpenLORIS-Object: A Robotic Vision Dataset and Benchmark for Lifelong Deep Learning](https://mlsanity.com/paper/1911.06487/) (2019)
- [Enabling On-Device CNN Training by Self-Supervised Instance Filtering and Error Map Pruning](https://mlsanity.com/paper/2007.03213/) (2020)
- [Intermittent Inference with Nonuniformly Compressed Multi-Exit Neural Network for Energy Harvesting Powered Devices](https://mlsanity.com/paper/2004.11293/) (2020)
- [Temporally Coherent Embeddings for Self-Supervised Video Representation Learning](https://mlsanity.com/paper/2004.02753/) (2020)
- [A Survey of Deep Learning Techniques for Mobile Robot Applications](https://mlsanity.com/paper/1803.07608/) (2018)

## Built on by (most cited first, 21 of 21)

- [Federated Contrastive Learning for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2204.10983/) (2022)
- [AutoReP: Automatic ReLU Replacement for Fast Private Network Inference](https://mlsanity.com/paper/2308.10134/) (2023)
- [LinGCN: Structural Linearized Graph Convolutional Network for Homomorphically Encrypted Inference](https://mlsanity.com/paper/2309.14331/) (2023)
- [Federated Contrastive Learning for Dermatological Disease Diagnosis via On-device Learning](https://mlsanity.com/paper/2202.07470/) (2022)
- [Unlocking Memorization in Large Language Models with Dynamic Soft Prompting](https://mlsanity.com/paper/2409.13853/) (2024)
- [Enabling On-Device Large Language Model Personalization with Self-Supervised Data Selection and Synthesis](https://mlsanity.com/paper/2311.12275/) (2023)
- [Distributed Contrastive Learning for Medical Image Segmentation](https://mlsanity.com/paper/2208.03808/) (2022)
- [Decentralized Unsupervised Learning of Visual Representations](https://mlsanity.com/paper/2111.10763/) (2021)
- [SmartFRZ: An Efficient Training Framework using Attention-Based Layer Freezing](https://mlsanity.com/paper/2401.16720/) (2024)
- [Exploration of Quantum Neural Architecture by Mixing Quantum Neuron Designs](https://mlsanity.com/paper/2109.03806/) (2021)
- [Training Machine Learning models at the Edge: A Survey](https://mlsanity.com/paper/2403.02619/) (2024)
- [BatchSampler: Sampling Mini-Batches for Contrastive Learning in Vision, Language, and Graphs](https://mlsanity.com/paper/2306.03355/) (2023)
- [Self-supervised On-device Federated Learning from Unlabeled Streams](https://mlsanity.com/paper/2212.01006/) (2022)
- [QuMoS: A Framework for Preserving Security of Quantum Machine Learning Model](https://mlsanity.com/paper/2304.11511/) (2023)
- [Federated Self-Supervised Contrastive Learning and Masked Autoencoder for Dermatological Disease Diagnosis](https://mlsanity.com/paper/2208.11278/) (2022)
- [All-in-One Tuning and Structural Pruning for Domain-Specific LLMs](https://mlsanity.com/paper/2412.14426/) (2024)
- [Edge-InversionNet: Enabling Efficient Inference of InversionNet on Edge Devices](https://mlsanity.com/paper/2310.09667/) (2023)
- [EdgeOL: Efficient in-situ Online Learning on Edge Devices](https://mlsanity.com/paper/2401.16694/) (2024)
- [A Self-guided Multimodal Approach to Enhancing Graph Representation Learning for Alzheimer's Diseases](https://mlsanity.com/paper/2412.06212/) (2024)
- [Enabling Weakly-Supervised Temporal Action Localization from On-Device Learning of the Video Stream](https://mlsanity.com/paper/2208.12673/) (2022)
- [Enabling On-Device Learning via Experience Replay with Efficient Dataset Condensation](https://mlsanity.com/paper/2405.16113/) (2024)

## Abstract (arXiv)

After a model is deployed on edge devices, it is desirable for these devices to learn from unlabeled data to continuously improve accuracy. Contrastive learning has demonstrated its great potential in learning from unlabeled data. However, the online input data are usually none independent and identically distributed (non-iid) and storages of edge devices are usually too limited to store enough representative data from different data classes. We propose a framework to automatically select the most representative data from the unlabeled input stream, which only requires a small data buffer for dynamic learning. Experiments show that accuracy and learning speed are greatly improved.