# Compressing VAE-Based Out-of-Distribution Detectors for Embedded Deployment

arXiv: https://arxiv.org/abs/2409.00880 · HTML: https://mlsanity.com/paper/2409.00880/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 26 references that are in the map.
- First posted on arXiv 2 Sep 2024.

## Builds on (most cited first, 13 of 13)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021)
- [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016)
- [A Unifying Review of Deep and Shallow Anomaly Detection](https://mlsanity.com/paper/2009.11732/) (2020)
- [Scalable Methods for 8-bit Training of Neural Networks](https://mlsanity.com/paper/1805.11046/) (2018)
- [q-Space Novelty Detection with Variational Autoencoders](https://mlsanity.com/paper/1806.02997/) (2018)
- [Efficient Out-of-Distribution Detection Using Latent Space of β-VAE for Cyber-Physical Systems](https://mlsanity.com/paper/2108.11800/) (2021)
- [Multi-Level Knowledge Distillation for Out-of-Distribution Detection in Text](https://mlsanity.com/paper/2211.11300/) (2022)
- [Improving Variational Autoencoder based Out-of-Distribution Detection for Embedded Real-time Applications](https://mlsanity.com/paper/2107.11750/) (2021)
- [Design Methodology for Deep Out-of-Distribution Detectors in Real-Time Cyber-Physical Systems](https://mlsanity.com/paper/2207.14694/) (2022)
- [Efficient Out-of-Distribution Detection of Melanoma with Wavelet-based Normalizing Flows](https://mlsanity.com/paper/2208.04639/) (2022)

## Abstract (arXiv)

Out-of-distribution (OOD) detectors can act as safety monitors in embedded cyber-physical systems by identifying samples outside a machine learning model's training distribution to prevent potentially unsafe actions. However, OOD detectors are often implemented using deep neural networks, which makes it difficult to meet real-time deadlines on embedded systems with memory and power constraints. We consider the class of variational autoencoder (VAE) based OOD detectors where OOD detection is performed in latent space, and apply quantization, pruning, and knowledge distillation. These techniques have been explored for other deep models, but no work has considered their combined effect on latent space OOD detection. While these techniques increase the VAE's test loss, this does not correspond to a proportional decrease in OOD detection performance and we leverage this to develop lean OOD detectors capable of real-time inference on embedded CPUs and GPUs. We propose a design methodology that combines all three compression techniques and yields a significant decrease in memory and execution time while maintaining AUROC for a given OOD detector. We demonstrate this methodology with two existing OOD detectors on a Jetson Nano and reduce GPU and CPU inference time by 20% and 28% respectively while keeping AUROC within 5% of the baseline.