# Teacher-free Latent Self-distillation and Class-separable Representations for Lightweight IoT Attack Detection

arXiv: https://arxiv.org/abs/2403.15509 · HTML: https://mlsanity.com/paper/2403.15509/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation.
- Builds on 13 of its 44 references that are in the map.
- First posted on arXiv 22 Mar 2024.

## Placed in

- Knowledge distillation / [Federated, robust and private distillation](https://mlsanity.com/distill/federated-robust-and-private-distillation/)

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [Preventing Posterior Collapse with delta-VAEs](https://mlsanity.com/paper/1901.03416/) (2019)
- [N-BaIoT: Network-based Detection of IoT Botnet Attacks Using Deep Autoencoders](https://mlsanity.com/paper/1805.03409/) (2018)
- [Preventing Oversmoothing in VAE via Generalized Variance Parameterization](https://mlsanity.com/paper/2102.08663/) (2021)
- [Self-Distilled Self-Supervised Representation Learning](https://mlsanity.com/paper/2111.12958/) (2021)
- [IGRF-RFE: A Hybrid Feature Selection Method for MLP-based Network Intrusion Detection on UNSW-NB15 Dataset](https://mlsanity.com/paper/2203.16365/) (2022)
- [Constrained Twin Variational Auto-Encoder for Intrusion Detection in IoT Systems](https://mlsanity.com/paper/2312.02490/) (2023)
- [The Role of Codeword-to-Class Assignments in Error-Correcting Codes: An Empirical Study](https://mlsanity.com/paper/2302.05334/) (2023)
- [Online Self-Supervised Deep Learning for Intrusion Detection Systems](https://mlsanity.com/paper/2306.13030/) (2023)

## Abstract (arXiv)

Knowledge distillation (KD) has been widely used to improve lightweight AI models by transferring soft-label knowledge from a large teacher model to a student model. However, existing KD methods are primarily designed for the image domain rather than lightweight IoT devices, and they often struggle to maintain well-separated feature representations for different attack types, especially as the number of classes increases and attack behaviors become more diverse. This paper proposes a novel textitteacher-free latent self-distillation framework based on a Twin Autoencoder (TAE). Instead of relying on an external teacher, TAE self-learns intrinsic class-wise latent representations, which act as soft labels, similar to KD, but without requiring a teacher model. The decoder then projects the input back into soft labels, enforcing class separation in the decoder output. The resulting decoder representations are used for classification, improving the discrimination between benign and malicious traffic while maintaining a lightweight design suitable for IoT deployment. We theoretically derive conditions for perfect class separation and show that lower empirical risk yields better representations, identifying regimes where TAE achieves strictly lower empirical risk than models using fixed class centers. Extensive experiments on 13 cybersecurity datasets, covering IoT botnets, network intrusion detection, malware, cloud DDoS, and synthetic multi-class data, show that TAE achieves up to 96.1% average accuracy for IoT attack detection and 98.7% for cloud intrusion detection. With a compact model size (1 MB) and ultra-fast inference (0.26 μs per sample), TAE offers a practical and scalable solution for real-world cybersecurity and IoT systems.