mlsanityData to 14 Oct 2026

Semi-Supervised Online Learning on the Edge by Transforming Knowledge from Teacher Models

arXiv 2512.16866 ↗First posted 18 Dec 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Edge machine learning (Edge ML) enables training ML models using the vast data distributed across network edges. However, many existing approaches assume static models trained centrally and then deployed, making them ineffective against unseen data. To address this, Online Edge ML allows models to be trained directly on edge devices and updated continuously with new data. This paper explores a key challenge of Online Edge ML: "How to determine labels for truly future, unseen data points". We propose Knowledge Transformation (KT), a hybrid method combining Knowledge Distillation, Active Learning, and causal reasoning. In short, KT acts as the oracle in active learning by transforming knowledge from a teacher model to generate pseudo-labels for training a student model. To verify the validity of the method, we conducted simulation experiments with two setups: (1) using a less stable teacher model and (2) a relatively more stable teacher model. Results indicate that when a stable teacher model is given, the student model can eventually reach its expected maximum performance. KT is potentially beneficial for scenarios that meet the following circumstances: (1) when the teacher's task is generic, which means existing pre-trained models might be adequate for its task, so there will be no need to train the teacher model from scratch; and/or (2) when the label for the student's task is difficult or expensive to acquire.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 43 references that are in the map. First posted on arXiv 18 Dec 2025.

Builds on

15 of 43 references.

References · 15

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  6. 2020Knowledge Distillation: A Surveycited by 1,603
  7. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,060
  8. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 664
  9. 2022On-Device Training Under 256KB Memorycited by 160
  10. 2018CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUscited by 121
  11. 2021TinyOL: TinyML with Online-Learning on Microcontrollerscited by 65
  12. 2020Deep Anomaly Detection for Time-series Data in Industrial IoT: A Communication-Efficient On-device Federated Learning Approachcited by 56
3 more
  1. 2019A Neural Network-Based On-device Learning Anomaly Detector for Edge Devicescited by 18
  2. 2022SemiPFL: Personalized Semi-Supervised Federated Learning Framework for Edge Intelligencecited by 7
  3. 2019Privacy is What We Care About: Experimental Investigation of Federated Learning on Edge Devicescited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Knowledge Distillation Meets Open-Set Semi-Supervised Learning
  2. 2022Parameter-Efficient and Student-Friendly Knowledge Distillation
  3. 2021Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework
  4. 2023Talking Models: Distill Pre-trained Knowledge to Downstream Models via Interactive Communication
  5. 2021Semi-Online Knowledge Distillation
  6. 2022Robust Active Distillation
  7. 2021Distilling a Powerful Student Model via Online Knowledge Distillation
  8. 2023Generalization Matters: Loss Minima Flattening via Parameter Hybridization for Efficient Online Knowledge Distillation