# DATE: Detecting Anomalies in Text via Self-Supervision of Transformers

arXiv: https://arxiv.org/abs/2104.05591 · HTML: https://mlsanity.com/paper/2104.05591/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 44 references that are in the map.
- First posted on arXiv 12 Apr 2021.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery](https://mlsanity.com/paper/1703.05921/) (2017)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [Deep Anomaly Detection Using Geometric Transformations](https://mlsanity.com/paper/1805.10917/) (2018)
- [BERTje: A Dutch BERT Model](https://mlsanity.com/paper/1912.09582/) (2019)

## Built on by (most cited first, 25 of 25)

- [ADBench: Anomaly Detection Benchmark](https://mlsanity.com/paper/2206.09426/) (2022)
- [TaxoCom: Topic Taxonomy Completion with Hierarchical Discovery of Novel Topic Clusters](https://mlsanity.com/paper/2201.06771/) (2022)
- [AD-LLM: Benchmarking Large Language Models for Anomaly Detection](https://mlsanity.com/paper/2412.11142/) (2024)
- [Self-Supervised Anomaly Detection in Computer Vision and Beyond: A Survey and Outlook](https://mlsanity.com/paper/2205.05173/) (2022)
- [NLP-ADBench: NLP Anomaly Detection Benchmark](https://mlsanity.com/paper/2412.04784/) (2024)
- [Multi-Level Knowledge Distillation for Out-of-Distribution Detection in Text](https://mlsanity.com/paper/2211.11300/) (2022)
- [Few-shot Anomaly Detection in Text with Deviation Learning](https://mlsanity.com/paper/2308.11780/) (2023)
- [Enhancing Out-of-Distribution Detection in Natural Language Understanding via Implicit Layer Ensemble](https://mlsanity.com/paper/2210.11034/) (2022)
- [See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language Balance to Mitigate Dominant Modality Bias](https://mlsanity.com/paper/2503.13834/) (2025)
- [R2D2: Robust Data-to-Text with Replacement Detection](https://mlsanity.com/paper/2205.12467/) (2022)
- [Self-Supervised Losses for One-Class Textual Anomaly Detection](https://mlsanity.com/paper/2204.05695/) (2022)
- [Text Anomaly Detection with Simplified Isolation Kernel](https://mlsanity.com/paper/2510.13197/) (2025)
- [FastLogAD: Log Anomaly Detection with Mask-Guided Pseudo Anomaly Generation and Discrimination](https://mlsanity.com/paper/2404.08750/) (2024)
- [Out-of-Distribution Detection through Soft Clustering with Non-Negative Kernel Regression](https://mlsanity.com/paper/2407.13141/) (2024)
- [Beyond a Single Perspective: Text Anomaly Detection with Multi-View Language Representations](https://mlsanity.com/paper/2601.17786/) (2026)
- [An Embarrassingly Simple Detector for Model Extraction Attacks in Large Language Model API Traffic](https://mlsanity.com/paper/2606.05725/) (2026)
- [Universal Domain Adaptation for Robust Handling of Distributional Shifts in NLP](https://mlsanity.com/paper/2310.14849/) (2023)
- [Out-of-Category Document Identification Using Target-Category Names as Weak Supervision](https://mlsanity.com/paper/2111.12796/) (2021)
- [A Robust Autoencoder Ensemble-Based Approach for Anomaly Detection in Text](https://mlsanity.com/paper/2405.13031/) (2024)
- [AR-Pro: Counterfactual Explanations for Anomaly Repair with Formal Properties](https://mlsanity.com/paper/2410.24178/) (2024)
- [Text-ADBench: Text Anomaly Detection Benchmark Based on LLM Embeddings](https://mlsanity.com/paper/2507.12295/) (2025)
- [Masked Diffusion Modeling for Anomaly Detection](https://mlsanity.com/paper/2605.30046/) (2026)
- [SIM: Subspace Interaction-based Method for Token-Level Text Anomaly Detection](https://mlsanity.com/paper/2609.08200/) (2026)
- [DiFA: Dual Evidence Fusion and Aggregation for Token-Level Text Anomaly Detection](https://mlsanity.com/paper/2609.22136/) (2026)
- [The Privacy Fallacy of Crowdsourced Fine-Tuning: Extracting Proprietary Data via Topic-Based Poisoning](https://mlsanity.com/paper/2609.33985/) (2026)

## Abstract (arXiv)

Leveraging deep learning models for Anomaly Detection (AD) has seen widespread use in recent years due to superior performances over traditional methods. Recent deep methods for anomalies in images learn better features of normality in an end-to-end self-supervised setting. These methods train a model to discriminate between different transformations applied to visual data and then use the output to compute an anomaly score. We use this approach for AD in text, by introducing a novel pretext task on text sequences. We learn our DATE model end-to-end, enforcing two independent and complementary self-supervision signals, one at the token-level and one at the sequence-level. Under this new task formulation, we show strong quantitative and qualitative results on the 20Newsgroups and AG News datasets. In the semi-supervised setting, we outperform state-of-the-art results by +13.5% and +6.9%, respectively (AUROC). In the unsupervised configuration, DATE surpasses all other methods even when 10% of its training data is contaminated with outliers (compared with 0% for the others).