# MAD-OOD: A Deep Learning Cluster-Driven Framework for an Out-of-Distribution Malware Detection and Classification

arXiv: https://arxiv.org/abs/2512.17594 · HTML: https://mlsanity.com/paper/2512.17594/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 19 Dec 2025.

## Builds on (most cited first, 19 of 19)

- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [Exploring the Limits of Out-of-Distribution Detection](https://mlsanity.com/paper/2106.03004/) (2021)
- [Deep Anomaly Detection Using Geometric Transformations](https://mlsanity.com/paper/1805.10917/) (2018)
- [OpenOOD: Benchmarking Generalized Out-of-Distribution Detection](https://mlsanity.com/paper/2210.07242/) (2022)
- [Rethinking Reconstruction Autoencoder-Based Out-of-Distribution Detection](https://mlsanity.com/paper/2203.02194/) (2022)
- [Multi-Class Data Description for Out-of-distribution Detection](https://mlsanity.com/paper/2104.00941/) (2021)
- [Out of Distribution Data Detection Using Dropout Bayesian Neural Networks](https://mlsanity.com/paper/2202.08985/) (2022)
- [Out-of-Distribution Data: An Acquaintance of Adversarial Examples -- A Survey](https://mlsanity.com/paper/2404.05219/) (2024)
- [AI Powered Anti-Cyber Bullying System using Machine Learning Algorithm of Multinomial Naive Bayes and Optimized Linear Support Vector Machine](https://mlsanity.com/paper/2207.11897/) (2022)
- [Adversarial Sampling for Fairness Testing in Deep Neural Network](https://mlsanity.com/paper/2303.02874/) (2023)
- [Performance Comparison and Implementation of Bayesian Variants for Network Intrusion Detection](https://mlsanity.com/paper/2308.11834/) (2023)
- [Encoder-Decoder Based Long Short-Term Memory (LSTM) Model for Video Captioning](https://mlsanity.com/paper/2401.02052/) (2023)
- [Implementation of Data Mining on a Secure Cloud Computing over a Web API using Supervised Machine Learning Algorithm](https://mlsanity.com/paper/2208.06433/) (2022)
- [Enhancing Border Security and Countering Terrorism Through Computer Vision: a Field of Artificial Intelligence](https://mlsanity.com/paper/2303.02869/) (2023)
- [Deep Learning-Based Speech and Vision Synthesis to Improve Phishing Attack Detection through a Multi-layer Adaptive Framework](https://mlsanity.com/paper/2402.17249/) (2024)
- [An Investigation into the Performances of the State-of-the-art Machine Learning Approaches for Various Cyber-attack Detection: A Survey](https://mlsanity.com/paper/2402.17045/) (2024)
- [Topology of Out-of-Distribution Examples in Deep Neural Networks](https://mlsanity.com/paper/2501.12522/) (2025)

## Abstract (arXiv)

Out of distribution (OOD) detection remains a critical challenge in malware classification due to the substantial intra family variability introduced by polymorphic and metamorphic malware variants. Most existing deep learning based malware detectors rely on closed world assumptions and fail to adequately model this intra class variation, resulting in degraded performance when confronted with previously unseen malware families. This paper presents MADOOD, a novel two stage, cluster driven deep learning framework for robust OOD malware detection and classification. In the first stage, malware family embeddings are modeled using class conditional spherical decision boundaries derived from Gaussian Discriminant Analysis (GDA), enabling statistically grounded separation of indistribution and OOD samples without requiring OOD data during training. Z score based distance analysis across multiple class centroids is employed to reliably identify anomalous samples in the latent space. In the second stage, a deep neural network integrates cluster based predictions, refined embeddings, and supervised classifier outputs to enhance final classification accuracy. Extensive evaluations on benchmark malware datasets comprising 25 known families and multiple novel OOD variants demonstrate that MADOOD significantly outperforms state of the art OOD detection methods, achieving an AUC of up to 0.911 on unseen malware families. The proposed framework provides a scalable, interpretable, and statistically principled solution for real world malware detection and anomaly identification in evolving cybersecurity environments.