# AD-LLM: Benchmarking Large Language Models for Anomaly Detection

arXiv: https://arxiv.org/abs/2412.11142 · HTML: https://mlsanity.com/paper/2412.11142/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 64 references that are in the map.
- First posted on arXiv 15 Dec 2024.

## Builds on (most cited first, 30 of 35)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [Specializing Smaller Language Models towards Multi-Step Reasoning](https://mlsanity.com/paper/2301.12726/) (2023)
- [On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey](https://mlsanity.com/paper/2406.15126/) (2024)
- [PyOD: A Python Toolbox for Scalable Outlier Detection](https://mlsanity.com/paper/1901.01588/) (2019)
- [Orca 2: Teaching Small Language Models How to Reason](https://mlsanity.com/paper/2311.11045/) (2023)
- [ECOD: Unsupervised Outlier Detection Using Empirical Cumulative Distribution Functions](https://mlsanity.com/paper/2201.00382/) (2022)
- [Deep Learning for Medical Anomaly Detection -- A Survey](https://mlsanity.com/paper/2012.02364/) (2020)
- [Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions](https://mlsanity.com/paper/2306.04140/) (2023)
- [Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Era](https://mlsanity.com/paper/2404.11457/) (2024)
- [Real-Time Anomaly Detection and Reactive Planning with Large Language Models](https://mlsanity.com/paper/2407.08735/) (2024)
- [Generative Adversarial Active Learning for Unsupervised Outlier Detection](https://mlsanity.com/paper/1809.10816/) (2018)
- [Towards Generic Anomaly Detection and Understanding: Large-scale Visual-linguistic Model (GPT-4V) Takes the Lead](https://mlsanity.com/paper/2311.02782/) (2023)
- [Interpretable Online Log Analysis Using Large Language Models with Prompt Strategies](https://mlsanity.com/paper/2308.07610/) (2023)
- [Do LLMs Understand Visual Anomalies? Uncovering LLM's Capabilities in Zero-shot Anomaly Detection](https://mlsanity.com/paper/2404.09654/) (2024)
- [Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell](https://mlsanity.com/paper/2406.14673/) (2024)
- [PyGOD: A Python Library for Graph Outlier Detection](https://mlsanity.com/paper/2204.12095/) (2022)
- [On the Importance of Building High-quality Training Datasets for Neural Code Search](https://mlsanity.com/paper/2202.06649/) (2022)
- [Large Language Models for Anomaly and Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2409.01980/) (2024)
- [Political-LLM: Large Language Models in Political Science](https://mlsanity.com/paper/2412.06864/) (2024)
- [DATE: Detecting Anomalies in Text via Self-Supervision of Transformers](https://mlsanity.com/paper/2104.05591/) (2021)
- [RDRec: Rationale Distillation for LLM-based Recommendation](https://mlsanity.com/paper/2405.10587/) (2024)
- [NLP-ADBench: NLP Anomaly Detection Benchmark](https://mlsanity.com/paper/2412.04784/) (2024)
- [PyOD 2: A Python Library for Outlier Detection with LLM-powered Model Selection](https://mlsanity.com/paper/2412.12154/) (2024)
- [MetaOOD: Automatic Selection of OOD Detection Models](https://mlsanity.com/paper/2410.03074/) (2024)
- [Data-Efficient Learning via Clustering-Based Sensitivity Sampling: Foundation Models and Beyond](https://mlsanity.com/paper/2402.17327/) (2024)
- [AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment](https://mlsanity.com/paper/2411.10606/) (2024)

## Built on by (most cited first, 26 of 26)

- [FORTIS: Benchmarking Over-Privilege in Agent Skills](https://mlsanity.com/paper/2605.09163/) (2026)
- [Can Multimodal LLMs Perform Time Series Anomaly Detection?](https://mlsanity.com/paper/2502.17812/) (2025)
- [Adaptive Fault Tolerance Mechanisms of Large Language Models in Cloud Computing Environments](https://mlsanity.com/paper/2503.12228/) (2025)
- [StealthRank: LLM Ranking Manipulation via Stealthy Prompt Optimization](https://mlsanity.com/paper/2504.05804/) (2025)
- [Scalability Optimization in Cloud-Based AI Inference Services: Strategies for Real-Time Load Balancing and Automated Scaling](https://mlsanity.com/paper/2504.15296/) (2025)
- [ClimateLLM: Efficient Weather Forecasting via Frequency-Aware Large Language Models](https://mlsanity.com/paper/2502.11059/) (2025)
- [The Autonomy Tax: Defense Training Breaks LLM Agents](https://mlsanity.com/paper/2603.19423/) (2026)
- [TAD-Bench: A Comprehensive Benchmark for Embedding-Based Text Anomaly Detection](https://mlsanity.com/paper/2501.11960/) (2025)
- [Discovering Knowledge Deficiencies of Language Models on Massive Knowledge Base](https://mlsanity.com/paper/2503.23361/) (2025)
- [MUSE: MCTS-Driven Red Teaming Framework for Enhanced Multi-Turn Dialogue Safety in Large Language Models](https://mlsanity.com/paper/2509.14651/) (2025)
- [Multimodal RAG-driven Anomaly Detection and Classification in Laser Powder Bed Fusion using Large Language Models](https://mlsanity.com/paper/2505.13828/) (2025)
- [CoAct: Co-Active LLM Preference Learning with Human-AI Synergy](https://mlsanity.com/paper/2604.17501/) (2026)
- [Towards Robust LLM Post-Training: Automatic Failure Management for Reinforcement Fine-Tuning](https://mlsanity.com/paper/2605.04431/) (2026)
- [Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias](https://mlsanity.com/paper/2607.11871/) (2026)
- [Detecting Voice Phishing with Precision: Fine-Tuning Small Language Models](https://mlsanity.com/paper/2506.06180/) (2025)
- [Text-ADBench: Text Anomaly Detection Benchmark Based on LLM Embeddings](https://mlsanity.com/paper/2507.12295/) (2025)
- [Cat-DPO: Category-Adaptive Safety Alignment](https://mlsanity.com/paper/2604.17299/) (2026)
- [Bridging Unsupervised and Semi-Supervised Anomaly Detection: A Theoretically-Grounded and Practical Framework with Synthetic Anomalies](https://mlsanity.com/paper/2506.13955/) (2025)
- [TAGFN: A Text-Attributed Graph Dataset for Fake News Detection in the Age of LLMs](https://mlsanity.com/paper/2511.21624/) (2025)
- [AuditCopilot: Leveraging LLMs for Fraud Detection in Double-Entry Bookkeeping](https://mlsanity.com/paper/2512.02726/) (2025)
- [Spatio-Temporal Graphs Beyond Grids: Benchmark for Maritime Anomaly Detection](https://mlsanity.com/paper/2512.20086/) (2025)
- [ChatAD: Reasoning-Enhanced Time-Series Anomaly Detection with Multi-Turn Instruction Evolution](https://mlsanity.com/paper/2601.13546/) (2026)
- [Detection of adversarial intent in Human-AI teams using LLMs](https://mlsanity.com/paper/2603.20976/) (2026)
- [Agent Safety Is Action Alignment](https://mlsanity.com/paper/2606.28739/) (2026)
- [GTA: Graph Theory Agent and Benchmark for Algorithmic Graph Reasoning with LLMs](https://mlsanity.com/paper/2609.12265/) (2026)
- [LLM4Trust: Exploring the Capabilities of Large Language Models for Trust Evaluation](https://mlsanity.com/paper/2609.33521/) (2026)

## Abstract (arXiv)

Anomaly detection (AD) is an important machine learning task with many real-world uses, including fraud detection, medical diagnosis, and industrial monitoring. Within natural language processing (NLP), AD helps detect issues like spam, misinformation, and unusual user activity. Although large language models (LLMs) have had a strong impact on tasks such as text generation and summarization, their potential in AD has not been studied enough. This paper introduces AD-LLM, the first benchmark that evaluates how LLMs can help with NLP anomaly detection. We examine three key tasks: (i) zero-shot detection, using LLMs' pre-trained knowledge to perform AD without tasks-specific training; (ii) data augmentation, generating synthetic data and category descriptions to improve AD models; and (iii) model selection, using LLMs to suggest unsupervised AD models. Through experiments with different datasets, we find that LLMs can work well in zero-shot AD, that carefully designed augmentation methods are useful, and that explaining model selection for specific datasets remains challenging. Based on these results, we outline six future research directions on LLMs for AD.