# Online Deep Learning based on Auto-Encoder

arXiv: https://arxiv.org/abs/2201.07383 · HTML: https://mlsanity.com/paper/2201.07383/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 54 references that are in the map.
- First posted on arXiv 19 Jan 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [A Structured Self-attentive Sentence Embedding](https://mlsanity.com/paper/1703.03130/) (2017)
- [Overcoming Catastrophic Forgetting by Incremental Moment Matching](https://mlsanity.com/paper/1703.08475/) (2017)
- [Online Deep Learning: Learning Deep Neural Networks on the Fly](https://mlsanity.com/paper/1711.03705/) (2017)
- [N-BaIoT: Network-based Detection of IoT Botnet Attacks Using Deep Autoencoders](https://mlsanity.com/paper/1805.03409/) (2018)
- [Spiking Neural Networks and Online Learning: An Overview and Perspectives](https://mlsanity.com/paper/1908.08019/) (2019)
- [Random Feature-based Online Multi-kernel Learning in Environments with Unknown Dynamics](https://mlsanity.com/paper/1712.09983/) (2017)
- [DEVDAN: Deep Evolving Denoising Autoencoder](https://mlsanity.com/paper/1910.04062/) (2019)
- [Autonomous Deep Learning: Continual Learning Approach for Dynamic Environments](https://mlsanity.com/paper/1810.07348/) (2018)
- [Improved and Scalable Online Learning of Spatial Concepts and Language Models with Mapping](https://mlsanity.com/paper/1803.03481/) (2018)
- [Reservoir of Diverse Adaptive Learners and Stacking Fast Hoeffding Drift Detection Methods for Evolving Data Streams](https://mlsanity.com/paper/1709.02457/) (2017)
- [Reconstruction Regularized Deep Metric Learning for Multi-label Image Classification](https://mlsanity.com/paper/2007.13547/) (2020)
- [Online Bayesian Passive-Aggressive Learning](https://mlsanity.com/paper/1312.3388/) (2013)
- [Online and Stable Learning of Analysis Operators](https://mlsanity.com/paper/1704.00227/) (2017)
- [Autonomous Deep Learning: Incremental Learning of Denoising Autoencoder for Evolving Data Streams](https://mlsanity.com/paper/1809.09081/) (2018)

## Abstract (arXiv)

Online learning is an important technical means for sketching massive real-time and high-speed data. Although this direction has attracted intensive attention, most of the literature in this area ignore the following three issues: (1) they think little of the underlying abstract hierarchical latent information existing in examples, even if extracting these abstract hierarchical latent representations is useful to better predict the class labels of examples; (2) the idea of preassigned model on unseen datapoints is not suitable for modeling streaming data with evolving probability distribution. This challenge is referred as model flexibility. And so, with this in minds, the online deep learning model we need to design should have a variable underlying structure; (3) moreover, it is of utmost importance to fusion these abstract hierarchical latent representations to achieve better classification performance, and we should give different weights to different levels of implicit representation information when dealing with the data streaming where the data distribution changes. To address these issues, we propose a two-phase Online Deep Learning based on Auto-Encoder (ODLAE). Based on auto-encoder, considering reconstruction loss, we extract abstract hierarchical latent representations of instances; Based on predictive loss, we devise two fusion strategies: the output-level fusion strategy, which is obtained by fusing the classification results of encoder each hidden layer; and feature-level fusion strategy, which is leveraged self-attention mechanism to fusion every hidden layer output. Finally, in order to improve the robustness of the algorithm, we also try to utilize the denoising auto-encoder to yield hierarchical latent representations. Experimental results on different datasets are presented to verify the validity of our proposed algorithm (ODLAE) outperforms several baselines.