# A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-based Variational Autoencoder

arXiv: https://arxiv.org/abs/1711.00614 · HTML: https://mlsanity.com/paper/1711.00614/

## Key facts

- Cited by 209 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Nov 2017.

## Built on by (most cited first, 30 of 209)

- [Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy](https://mlsanity.com/paper/2110.02642/) (2021)
- [Graph Neural Network-Based Anomaly Detection in Multivariate Time Series](https://mlsanity.com/paper/2106.06947/) (2021)
- [TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data](https://mlsanity.com/paper/2201.07284/) (2022)
- [A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detection](https://mlsanity.com/paper/2307.03759/) (2023)
- [Deep Learning for Time Series Anomaly Detection: A Survey](https://mlsanity.com/paper/2211.05244/) (2022)
- [Learning Graph Structures with Transformer for Multivariate Time Series Anomaly Detection in IoT](https://mlsanity.com/paper/2104.03466/) (2021)
- [DCdetector: Dual Attention Contrastive Representation Learning for Time Series Anomaly Detection](https://mlsanity.com/paper/2306.10347/) (2023)
- [Towards a Rigorous Evaluation of Time-series Anomaly Detection](https://mlsanity.com/paper/2109.05257/) (2021)
- [An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series](https://mlsanity.com/paper/2109.11428/) (2021)
- [A Survey on Time-Series Pre-Trained Models](https://mlsanity.com/paper/2305.10716/) (2023)
- [TFAD: A Decomposition Time Series Anomaly Detection Architecture with Time-Frequency Analysis](https://mlsanity.com/paper/2210.09693/) (2022)
- [A survey of multimodal deep generative models](https://mlsanity.com/paper/2207.02127/) (2022)
- [Neural Contextual Anomaly Detection for Time Series](https://mlsanity.com/paper/2107.07702/) (2021)
- [ImDiffusion: Imputed Diffusion Models for Multivariate Time Series Anomaly Detection](https://mlsanity.com/paper/2307.00754/) (2023)
- [Towards a General Time Series Anomaly Detector with Adaptive Bottlenecks and Dual Adversarial Decoders](https://mlsanity.com/paper/2405.15273/) (2024)
- [Time Series Anomaly Detection for Cyber-Physical Systems via Neural System Identification and Bayesian Filtering](https://mlsanity.com/paper/2106.07992/) (2021)
- [Large language models can be zero-shot anomaly detectors for time series?](https://mlsanity.com/paper/2405.14755/) (2024)
- [A Survey on Anomaly Detection for Technical Systems using LSTM Networks](https://mlsanity.com/paper/2105.13810/) (2021)
- [MEMTO: Memory-guided Transformer for Multivariate Time Series Anomaly Detection](https://mlsanity.com/paper/2312.02530/) (2023)
- [Large Language Model Guided Knowledge Distillation for Time Series Anomaly Detection](https://mlsanity.com/paper/2401.15123/) (2024)
- [Unsupervised Model Selection for Time-series Anomaly Detection](https://mlsanity.com/paper/2210.01078/) (2022)
- [AnomalyBERT: Self-Supervised Transformer for Time Series Anomaly Detection using Data Degradation Scheme](https://mlsanity.com/paper/2305.04468/) (2023)
- [Navigating the Metric Maze: A Taxonomy of Evaluation Metrics for Anomaly Detection in Time Series](https://mlsanity.com/paper/2303.01272/) (2023)
- [DASVDD: Deep Autoencoding Support Vector Data Descriptor for Anomaly Detection](https://mlsanity.com/paper/2106.05410/) (2021)
- [MST-GAT: A Multimodal Spatial-Temporal Graph Attention Network for Time Series Anomaly Detection](https://mlsanity.com/paper/2310.11169/) (2023)
- [Regularization-based Continual Learning for Anomaly Detection in Discrete Manufacturing](https://mlsanity.com/paper/2101.00509/) (2021)
- [Do Deep Neural Networks Contribute to Multivariate Time Series Anomaly Detection?](https://mlsanity.com/paper/2204.01637/) (2022)
- [LSTM-Autoencoder based Anomaly Detection for Indoor Air Quality Time Series Data](https://mlsanity.com/paper/2204.06701/) (2022)
- [Correlation-aware Spatial-Temporal Graph Learning for Multivariate Time-series Anomaly Detection](https://mlsanity.com/paper/2307.08390/) (2023)
- [Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration](https://mlsanity.com/paper/2108.06038/) (2021)

## Abstract (arXiv)

The detection of anomalous executions is valuable for reducing potential hazards in assistive manipulation. Multimodal sensory signals can be helpful for detecting a wide range of anomalies. However, the fusion of high-dimensional and heterogeneous modalities is a challenging problem. We introduce a long short-term memory based variational autoencoder (LSTM-VAE) that fuses signals and reconstructs their expected distribution. We also introduce an LSTM-VAE-based detector using a reconstruction-based anomaly score and a state-based threshold. For evaluations with 1,555 robot-assisted feeding executions including 12 representative types of anomalies, our detector had a higher area under the receiver operating characteristic curve (AUC) of 0.8710 than 5 other baseline detectors from the literature. We also show the multimodal fusion through the LSTM-VAE is effective by comparing our detector with 17 raw sensory signals versus 4 hand-engineered features.