Text-ADBench: Text Anomaly Detection Benchmark Based on LLM Embeddings
Not on the map. Why?
Abstract (arXiv)
Text anomaly detection is a critical task in natural language processing (NLP), with applications spanning fraud detection, misinformation identification, spam detection and content moderation, etc. Despite significant advances in large language models (LLMs) and anomaly detection algorithms, the absence of standardized and comprehensive benchmarks for evaluating the existing anomaly detection methods on text data limits rigorous comparison and development of innovative approaches. This work performs a comprehensive empirical study and introduces a benchmark for text anomaly detection, leveraging embeddings from diverse pre-trained language models across a wide array of text datasets. Our work systematically evaluates the effectiveness of embedding-based text anomaly detection by incorporating (1) early language models (GloVe, BERT); (2) multiple LLMs (LLaMA-2, LLaMA-3, Mistral, OpenAI embedding models (small, ada, large)); (3) multi-domain text datasets (news, social media, scientific publications); (4) comprehensive evaluation metrics (AUROC, AUPRC). Our experiments reveal a critical empirical insight: embedding quality significantly governs anomaly detection efficacy, and deep learning-based approaches demonstrate no performance advantage over conventional shallow algorithms (e.g., KNN, OCSVM) when leveraging LLM-derived embeddings. In addition, we observe strongly low-rank characteristics in cross-model performance matrices, which enables an efficient strategy for rapid model evaluation (or embedding evaluation) and selection in practical applications. Furthermore, by open-sourcing our benchmark toolkit that includes all embeddings from different models and code, this work provides a foundation for future research in robust and scalable text anomaly detection systems. Our code repository is at https://github.com/jicongfan/Text-Anomaly-Detection-Benchmark.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 42 of its 94 references that are in the map. First posted on arXiv 16 Jul 2025.
Builds on
42 of 94 references.
References · 42
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
- 2018Deep contextualized word representationscited by 2,290
- 2018Deep Anomaly Detection with Outlier Exposurecited by 942
- 2016Bag of Tricks for Efficient Text Classificationcited by 876
- 2023A Comprehensive Overview of Large Language Modelscited by 648
- 2017Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discoverycited by 566
- 2020A Unifying Review of Deep and Shallow Anomaly Detectioncited by 425
30 more
- 2018GANomaly: Semi-Supervised Anomaly Detection via Adversarial Trainingcited by 409
- 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
- 2023Improving Text Embeddings with Large Language Modelscited by 303
- 2019Deep Semi-Supervised Anomaly Detectioncited by 291
- 2019OCGAN: One-class Novelty Detection Using GANs with Constrained Latent Representationscited by 199
- 2023AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Modelscited by 171
- 2019Deep Anomaly Detection with Deviation Networkscited by 153
- 2022ECOD: Unsupervised Outlier Detection Using Empirical Cumulative Distribution Functionscited by 153
- 2022Deep Learning for Time Series Anomaly Detection: A Surveycited by 131
- 2020Deep Learning for Medical Anomaly Detection -- A Surveycited by 127
- 2020DROCC: Deep Robust One-Class Classificationcited by 121
- 2017HDLTex: Hierarchical Deep Learning for Text Classificationcited by 106
- 2021Neural Transformation Learning for Deep Anomaly Detection Beyond Imagescited by 99
- 2022Deep Isolation Forest for Anomaly Detectioncited by 92
- 2018Learning Representations of Ultrahigh-dimensional Data for Random Distance-based Outlier Detectioncited by 79
- 2019Deep Weakly-supervised Anomaly Detectioncited by 59
- 2024Large Language Models for Forecasting and Anomaly Detection: A Systematic Literature Reviewcited by 53
- 2021Feature Encoding with AutoEncoders for Weakly-supervised Anomaly Detectioncited by 42
- 2017Unsupervised and Semi-supervised Anomaly Detection with LSTM Neural Networkscited by 40
- 2024AD-LLM: Benchmarking Large Language Models for Anomaly Detectioncited by 26
- 2021DATE: Detecting Anomalies in Text via Self-Supervision of Transformerscited by 25
- 2022Perturbation Learning Based Anomaly Detectioncited by 24
- 2024Can LLMs Serve As Time Series Anomaly Detectors?cited by 22
- 2024Networking Systems for Video Anomaly Detection: A Tutorial and Surveycited by 21
- 2024NLP-ADBench: NLP Anomaly Detection Benchmarkcited by 17
- 2019Factor Group-Sparse Regularization for Efficient Low-Rank Matrix Recoverycited by 15
- 2024Anomaly Detection of Tabular Data Using LLMscited by 12
- 2023Few-shot Anomaly Detection in Text with Deviation Learningcited by 11
- 2025TAD-Bench: A Comprehensive Benchmark for Embedding-Based Text Anomaly Detectioncited by 6
- 2024A Robust Autoencoder Ensemble-Based Approach for Anomaly Detection in Textcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.