mlsanityData to 14 Oct 2026

Self-Supervised Losses for One-Class Textual Anomaly Detection

arXiv 2204.05695 ↗First posted 12 Apr 2022cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Current deep learning methods for anomaly detection in text rely on supervisory signals in inliers that may be unobtainable or bespoke architectures that are difficult to tune. We study a simpler alternative: fine-tuning Transformers on the inlier data with self-supervised objectives and using the losses as an anomaly score. Overall, the self-supervision approach outperforms other methods under various anomaly detection scenarios, improving the AUROC score on semantic anomalies by 11.6% and on syntactic anomalies by 22.8% on average. Additionally, the optimal objective and resultant learnt representation depend on the type of downstream anomaly. The separability of anomalies and inliers signals that a representation is more effective for detecting semantic anomalies, whilst the presence of narrow feature directions signals a representation that is effective for detecting syntactic anomalies.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 24 of its 37 references that are in the map. First posted on arXiv 12 Apr 2022.

Builds on

24 of 37 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  4. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  6. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  7. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
  8. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  9. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
  10. 2020CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instancescited by 441
  11. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 294
  12. 2021Unsolved Problems in ML Safetycited by 292
12 more
  1. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  2. 2021SSD: A Unified Framework for Self-Supervised Outlier Detectioncited by 264
  3. 2020Contrastive Training for Improved Out-of-Distribution Detectioncited by 186
  4. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 177
  5. 2021Types of Out-of-Distribution Texts and How to Detect Themcited by 80
  6. 2021Contrastive Out-of-Distribution Detection for Pretrained Transformerscited by 71
  7. 2021Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detectioncited by 70
  8. 2021Towards Few-Shot Fact-Checking via Perplexitycited by 63
  9. 2020Where Are the Facts? Searching for Fact-checked Information to Alleviate the Spread of Fake Newscited by 48
  10. 2021DATE: Detecting Anomalies in Text via Self-Supervision of Transformerscited by 25
  11. 2021kFolden: k-Fold Ensemble for Out-Of-Distribution Detectioncited by 10
  12. 2021Brittle Features May Help Anomaly Detectioncited by 2

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 4.

5 papers.

  1. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
  2. 2022Multi-Level Knowledge Distillation for Out-of-Distribution Detection in Textcited by 11
  3. 2023Few-shot Anomaly Detection in Text with Deviation Learningcited by 11
  4. 2023Understanding the limitations of self-supervised learning for tabular anomaly detectioncited by 3
  5. 2023Estimating Semantic Similarity between In-Domain and Out-of-Domain Samplescited by 1