# Hybridization of Capsule and LSTM Networks for unsupervised anomaly detection on multivariate data

arXiv: https://arxiv.org/abs/2202.05538 · HTML: https://mlsanity.com/paper/2202.05538/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 73 references that are in the map.
- First posted on arXiv 11 Feb 2022.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [A review on outlier/anomaly detection in time series data](https://mlsanity.com/paper/2002.04236/) (2020)
- [A Deep Neural Network for Unsupervised Anomaly Detection and Diagnosis in Multivariate Time Series Data](https://mlsanity.com/paper/1811.08055/) (2018)
- [Evaluating Real-time Anomaly Detection Algorithms - the Numenta Anomaly Benchmark](https://mlsanity.com/paper/1510.03336/) (2015)
- [Unsupervised and Semi-supervised Anomaly Detection with LSTM Neural Networks](https://mlsanity.com/paper/1710.09207/) (2017)
- [Multivariate Industrial Time Series with Cyber-Attack Simulation: Fault Detection Using an LSTM-based Predictive Data Model](https://mlsanity.com/paper/1612.06676/) (2016)
- [No Routing Needed Between Capsules](https://mlsanity.com/paper/2001.09136/) (2020)
- [Brain Tumor Type Classification via Capsule Networks](https://mlsanity.com/paper/1802.10200/) (2018)
- [Unsupervised Motion Representation Learning with Capsule Autoencoders](https://mlsanity.com/paper/2110.00529/) (2021)
- [Forecasting Transportation Network Speed Using Deep Capsule Networks with Nested LSTM Models](https://mlsanity.com/paper/1811.04745/) (2018)
- [Homogeneous Vector Capsules Enable Adaptive Gradient Descent in Convolutional Neural Networks](https://mlsanity.com/paper/1906.08676/) (2019)

## Built on by (most cited first, 2 of 2)

- [Detection of Anomalies in Multivariate Time Series Using Ensemble Techniques](https://mlsanity.com/paper/2308.03171/) (2023)
- [Internet of Things Fault Detection and Classification via Multitask Learning](https://mlsanity.com/paper/2307.01234/) (2023)

## Abstract (arXiv)

Deep learning techniques have recently shown promise in the field of anomaly detection, providing a flexible and effective method of modelling systems in comparison to traditional statistical modelling and signal processing-based methods. However, there are a few well publicised issues Neural Networks (NN)s face such as generalisation ability, requiring large volumes of labelled data to be able to train effectively and understanding spatial context in data. This paper introduces a novel NN architecture which hybridises the Long-Short-Term-Memory (LSTM) and Capsule Networks into a single network in a branched input Autoencoder architecture for use on multivariate time series data. The proposed method uses an unsupervised learning technique to overcome the issues with finding large volumes of labelled training data. Experimental results show that without hyperparameter optimisation, using Capsules significantly reduces overfitting and improves the training efficiency. Additionally, results also show that the branched input models can learn multivariate data more consistently with or without Capsules in comparison to the non-branched input models. The proposed model architecture was also tested on an open-source benchmark, where it achieved state-of-the-art performance in outlier detection, and overall performs best over the metrics tested in comparison to current state-of-the art methods.