mlsanityData to 14 Oct 2026

Knowledge Distillation for Reservoir-based Classifier: Human Activity Recognition

arXiv 2505.22985 ↗First posted 29 May 2025cs.LG cs.AI cs.PFcited by 0

On the map: Knowledge distillation → Distillation for medical imaging

Abstract (arXiv)

This paper aims to develop an energy-efficient classifier for time-series data by introducing PatchEchoClassifier, a novel model that leverages a reservoir-based mechanism known as the Echo State Network (ESN). The model is designed for human activity recognition (HAR) using one-dimensional sensor signals and incorporates a tokenizer to extract patch-level representations. To train the model efficiently, we propose a knowledge distillation framework that transfers knowledge from a high-capacity MLP-Mixer teacher to the lightweight reservoir-based student model. Experimental evaluations on multiple HAR datasets demonstrate that our model achieves over 80 percent accuracy while significantly reducing computational cost. Notably, PatchEchoClassifier requires only about one-sixth of the floating point operations (FLOPS) compared to DeepConvLSTM, a widely used convolutional baseline. These results suggest that PatchEchoClassifier is a promising solution for real-time and energy-efficient human activity recognition in edge computing environments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation. Builds on 11 of its 26 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

11 of 26 references.

References · 11

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  6. 2019Relational Knowledge Distillationcited by 889
  7. 2023CrossKD: Cross-Head Knowledge Distillation for Object Detectioncited by 52
  8. 2021Improving Deep Learning for HAR with shallow LSTMscited by 30
  9. 2023High Speed Human Action Recognition using a Photonic Reservoir Computercited by 5
  10. 2023Too Good To Be True: performance overestimation in (re)current practices for Human Activity Recognitioncited by 2
  11. 2024Neuromorphic on-chip reservoir computing with spiking neural network architecturescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021A Light-weight Deep Human Activity Recognition Algorithm Using Multi-knowledge Distillation
  2. 2022Ultra-compact Binary Neural Networks for Human Activity Recognition on RISC-V Processors
  3. 2025Smooth-Distill: A Self-distillation Framework for Multitask Learning with Wearable Sensor Data
  4. 2021AHAR: Adaptive CNN for Energy-efficient Human Activity Recognition in Low-power Edge Devices
  5. 2022Human Activity Recognition on Microcontrollers with Quantized and Adaptive Deep Neural Networks
  6. 2026RAG-HAR+: Towards Cost-Efficient LLM-Based Human Activity Recognition for Edge Deployment
  7. 2021Cross-modal Knowledge Distillation for Vision-to-Sensor Action Recognition
  8. 2022TASKED: Transformer-based Adversarial learning for human activity recognition using wearable sensors via Self-KnowledgE Distillation