mlsanityData to 7 Oct 2026

DATE: Detecting Anomalies in Text via Self-Supervision of Transformers

arXiv 2104.05591 ↗First posted 12 Apr 2021cs.CLcited by 25

Not on the map. Why?

Abstract (arXiv)

Leveraging deep learning models for Anomaly Detection (AD) has seen widespread use in recent years due to superior performances over traditional methods. Recent deep methods for anomalies in images learn better features of normality in an end-to-end self-supervised setting. These methods train a model to discriminate between different transformations applied to visual data and then use the output to compute an anomaly score. We use this approach for AD in text, by introducing a novel pretext task on text sequences. We learn our DATE model end-to-end, enforcing two independent and complementary self-supervision signals, one at the token-level and one at the sequence-level. Under this new task formulation, we show strong quantitative and qualitative results on the 20Newsgroups and AG News datasets. In the semi-supervised setting, we outperform state-of-the-art results by +13.5% and +6.9%, respectively (AUROC). In the unsupervised configuration, DATE surpasses all other methods even when 10% of its training data is contaminated with outliers (compared with 0% for the others).

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 14 of its 44 references that are in the map. First posted on arXiv 12 Apr 2021.

Builds on

14 of 44 references.

References · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  8. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  9. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  10. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  11. 2017Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discoverycited by 566
  12. 2019CamemBERT: a Tasty French Language Modelcited by 368
2 more
  1. 2018Deep Anomaly Detection Using Geometric Transformationscited by 269
  2. 2019BERTje: A Dutch BERT Modelcited by 109

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 2; 2024: 6; 2025: 3; 2026: 6.

25 papers.

Sort:

By area:

  1. 2022ADBench: Anomaly Detection Benchmarkcited by 230
  2. 2022TaxoCom: Topic Taxonomy Completion with Hierarchical Discovery of Novel Topic Clusterscited by 30
  3. 2024AD-LLM: Benchmarking Large Language Models for Anomaly Detectioncited by 25
  4. 2022Self-Supervised Anomaly Detection in Computer Vision and Beyond: A Survey and Outlookcited by 24
  5. 2024NLP-ADBench: NLP Anomaly Detection Benchmarkcited by 17
  6. 2022Multi-Level Knowledge Distillation for Out-of-Distribution Detection in Textcited by 11
  7. 2023Few-shot Anomaly Detection in Text with Deviation Learningcited by 11
  8. 2022Enhancing Out-of-Distribution Detection in Natural Language Understanding via Implicit Layer Ensemblecited by 9
  9. 2025See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language Balance to Mitigate Dominant Modality Biascited by 9
  10. 2022R2D2: Robust Data-to-Text with Replacement Detectioncited by 8
  11. 2022Self-Supervised Losses for One-Class Textual Anomaly Detectioncited by 5
  12. 2025Text Anomaly Detection with Simplified Isolation Kernelcited by 5
13 more
  1. 2024FastLogAD: Log Anomaly Detection with Mask-Guided Pseudo Anomaly Generation and Discriminationcited by 4
  2. 2024Out-of-Distribution Detection through Soft Clustering with Non-Negative Kernel Regressioncited by 3
  3. 2026Beyond a Single Perspective: Text Anomaly Detection with Multi-View Language Representationscited by 3
  4. 2023Universal Domain Adaptation for Robust Handling of Distributional Shifts in NLPcited by 2
  5. 2021Out-of-Category Document Identification Using Target-Category Names as Weak Supervisioncited by 1
  6. 2024A Robust Autoencoder Ensemble-Based Approach for Anomaly Detection in Textcited by 1
  7. 2024AR-Pro: Counterfactual Explanations for Anomaly Repair with Formal Propertiescited by 1
  8. 2025Text-ADBench: Text Anomaly Detection Benchmark Based on LLM Embeddingscited by 1
  9. 2026An Embarrassingly Simple Detector for Model Extraction Attacks in Large Language Model API Trafficcited by 1
  10. 2026Masked Diffusion Modeling for Anomaly Detectioncited by 0
  11. 2026SIM: Subspace Interaction-based Method for Token-Level Text Anomaly Detectioncited by 0
  12. 2026DiFA: Dual Evidence Fusion and Aggregation for Token-Level Text Anomaly Detectioncited by 0
  13. 2026The Privacy Fallacy of Crowdsourced Fine-Tuning: Extracting Proprietary Data via Topic-Based Poisoningcited by 0