# Text-ADBench: Text Anomaly Detection Benchmark Based on LLM Embeddings

arXiv: https://arxiv.org/abs/2507.12295 · HTML: https://mlsanity.com/paper/2507.12295/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 94 references that are in the map.
- First posted on arXiv 16 Jul 2025.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery](https://mlsanity.com/paper/1703.05921/) (2017)
- [A Unifying Review of Deep and Shallow Anomaly Detection](https://mlsanity.com/paper/2009.11732/) (2020)
- [GANomaly: Semi-Supervised Anomaly Detection via Adversarial Training](https://mlsanity.com/paper/1805.06725/) (2018)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [Improving Text Embeddings with Large Language Models](https://mlsanity.com/paper/2401.00368/) (2023)
- [Deep Semi-Supervised Anomaly Detection](https://mlsanity.com/paper/1906.02694/) (2019)
- [OCGAN: One-class Novelty Detection Using GANs with Constrained Latent Representations](https://mlsanity.com/paper/1903.08550/) (2019)
- [AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models](https://mlsanity.com/paper/2308.15366/) (2023)
- [Deep Anomaly Detection with Deviation Networks](https://mlsanity.com/paper/1911.08623/) (2019)
- [ECOD: Unsupervised Outlier Detection Using Empirical Cumulative Distribution Functions](https://mlsanity.com/paper/2201.00382/) (2022)
- [Deep Learning for Time Series Anomaly Detection: A Survey](https://mlsanity.com/paper/2211.05244/) (2022)
- [Deep Learning for Medical Anomaly Detection -- A Survey](https://mlsanity.com/paper/2012.02364/) (2020)
- [DROCC: Deep Robust One-Class Classification](https://mlsanity.com/paper/2002.12718/) (2020)
- [HDLTex: Hierarchical Deep Learning for Text Classification](https://mlsanity.com/paper/1709.08267/) (2017)
- [Neural Transformation Learning for Deep Anomaly Detection Beyond Images](https://mlsanity.com/paper/2103.16440/) (2021)
- [Deep Isolation Forest for Anomaly Detection](https://mlsanity.com/paper/2206.06602/) (2022)
- [Learning Representations of Ultrahigh-dimensional Data for Random Distance-based Outlier Detection](https://mlsanity.com/paper/1806.04808/) (2018)
- [Deep Weakly-supervised Anomaly Detection](https://mlsanity.com/paper/1910.13601/) (2019)
- [Large Language Models for Forecasting and Anomaly Detection: A Systematic Literature Review](https://mlsanity.com/paper/2402.10350/) (2024)
- [Feature Encoding with AutoEncoders for Weakly-supervised Anomaly Detection](https://mlsanity.com/paper/2105.10500/) (2021)

## Built on by (most cited first, 1 of 1)

- [Rethinking Weak Supervision in Anomaly Detection: A Comprehensive Benchmark](https://mlsanity.com/paper/2605.26068/) (2026)

## Abstract (arXiv)

Text anomaly detection is a critical task in natural language processing (NLP), with applications spanning fraud detection, misinformation identification, spam detection and content moderation, etc. Despite significant advances in large language models (LLMs) and anomaly detection algorithms, the absence of standardized and comprehensive benchmarks for evaluating the existing anomaly detection methods on text data limits rigorous comparison and development of innovative approaches. This work performs a comprehensive empirical study and introduces a benchmark for text anomaly detection, leveraging embeddings from diverse pre-trained language models across a wide array of text datasets. Our work systematically evaluates the effectiveness of embedding-based text anomaly detection by incorporating (1) early language models (GloVe, BERT); (2) multiple LLMs (LLaMA-2, LLaMA-3, Mistral, OpenAI embedding models (small, ada, large)); (3) multi-domain text datasets (news, social media, scientific publications); (4) comprehensive evaluation metrics (AUROC, AUPRC). Our experiments reveal a critical empirical insight: embedding quality significantly governs anomaly detection efficacy, and deep learning-based approaches demonstrate no performance advantage over conventional shallow algorithms (e.g., KNN, OCSVM) when leveraging LLM-derived embeddings. In addition, we observe strongly low-rank characteristics in cross-model performance matrices, which enables an efficient strategy for rapid model evaluation (or embedding evaluation) and selection in practical applications. Furthermore, by open-sourcing our benchmark toolkit that includes all embeddings from different models and code, this work provides a foundation for future research in robust and scalable text anomaly detection systems. Our code repository is at https://github.com/jicongfan/Text-Anomaly-Detection-Benchmark.