mlsanityData to 14 Oct 2026

Transformaly -- Two (Feature Spaces) Are Better Than One

arXiv 2112.04185 ↗First posted 8 Dec 2021cs.CV cs.LGcited by 25

On the map: Knowledge distillation → Detection and dense prediction distillation

Abstract (arXiv)

Anomaly detection is a well-established research area that seeks to identify samples outside of a predetermined distribution. An anomaly detection pipeline is comprised of two main stages: (1) feature extraction and (2) normality score assignment. Recent papers used pre-trained networks for feature extraction achieving state-of-the-art results. However, the use of pre-trained networks does not fully-utilize the normal samples that are available at train time. This paper suggests taking advantage of this information by using teacher-student training. In our setting, a pretrained teacher network is used to train a student network on the normal training samples. Since the student network is trained only on normal samples, it is expected to deviate from the teacher network in abnormal cases. This difference can serve as a complementary representation to the pre-trained feature vector. Our method -- Transformaly -- exploits a pre-trained Vision Transformer (ViT) to extract both feature vectors: the pre-trained (agnostic) features and the teacher-student (fine-tuned) features. We report state-of-the-art AUROC results in both the common unimodal setting, where one class is considered normal and the rest are considered abnormal, and the multimodal setting, where all classes but one are considered normal, and just one class is considered abnormal. The code is available at https://github.com/MatanCohen1/Transformaly.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation. Builds on 29 of its 49 references that are in the map. First posted on arXiv 8 Dec 2021.

Builds on

29 of 49 references.

References · 29

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  6. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  7. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,328
  8. 2018Deep Anomaly Detection with Outlier Exposurecited by 942
  9. 2021ImageNet-21K Pretraining for the Massescited by 613
  10. 2021CutPaste: Self-Supervised Learning for Anomaly Detection and Localizationcited by 531
  11. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
  12. 2020CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instancescited by 441
17 more
  1. 2019Uninformed Students: Student-Teacher Anomaly Detection with Discriminative Latent Embeddingscited by 386
  2. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 316
  3. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 294
  4. 2018Deep Anomaly Detection Using Geometric Transformationscited by 269
  5. 2021VT-ADL: A Vision Transformer Network for Image Anomaly Detection and Localizationcited by 223
  6. 2020Classification-Based Anomaly Detection for General Datacited by 211
  7. 2020PANDA: Adapting Pretrained Features for Anomaly Detection and Segmentationcited by 175
  8. 2020Learning and Evaluating Representations for Deep One-class Classificationcited by 133
  9. 2018Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifierscited by 126
  10. 2018Efficient GAN-Based Anomaly Detectioncited by 122
  11. 2020Deep Nearest Neighbor Anomaly Detectioncited by 104
  12. 2021Inpainting Transformer for Anomaly Detectioncited by 91
  13. 2021Mean-Shifted Contrastive Loss for Anomaly Detectioncited by 88
  14. 2018Anomaly Detection with Generative Adversarial Networks for Multivariate Time Seriescited by 55
  15. 2019Detecting semantic anomaliescited by 50
  16. 2020Puzzle-AE: Novelty Detection in Images through Solving Puzzlescited by 23
  17. 2021Do We Really Need to Learn Representations from In-domain Data for Outlier Detection?cited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 10; 2024: 6; 2025: 5; 2026: 1.

25 papers.

Sort:

By area:

  1. 2023Improving neural network representations using human similarity judgmentscited by 38
  2. 2024GeneralAD: Anomaly Detection Across Domains by Attending to Distorted Featurescited by 29
  3. 2023Set Features for Fine-grained Anomaly Detectioncited by 13
  4. 2025PatchGuard: Adversarially Robust Anomaly Detection and Localization through Vision Transformers and Pseudo Anomaliescited by 8
  5. 2025Foundation Models and Transformers for Anomaly Detection: A Surveycited by 8
  6. 2023Unveiling the Unseen: A Comprehensive Survey on Explainable Anomaly Detection in Images and Videoscited by 7
  7. 2023No Free Lunch: The Hazards of Over-Expressive Representations in Anomaly Detectioncited by 7
  8. 2023A graph neural network-based model with Out-of-Distribution Robustness for enhancing Antiretroviral Therapy Outcome Prediction for HIV-1cited by 7
  9. 2025RODEO: Robust Outlier Detection via Exposing Adaptive Out-of-Distribution Samplescited by 7
  10. 2024Adversarially Robust Out-of-Distribution Detection Using Lyapunov-Stabilized Embeddingscited by 6
  11. 2022Fake It Till You Make It: Towards Accurate Near-Distribution Novelty Detectioncited by 4
  12. 2025Killing it with Zero-Shot: Adversarially Robust Novelty Detectioncited by 4
13 more
  1. 2022Positive Difference Distribution for Image Outlier Detection using Normalizing Flows and Contrastive Datacited by 3
  2. 2023Dissolving Is Amplifying: Towards Fine-Grained Anomaly Detectioncited by 3
  3. 2023Set Features for Anomaly Detectioncited by 3
  4. 2023Understanding normalization in contrastive representation learning and out-of-distribution detectioncited by 2
  5. 2024Critical Review for One-class Classification: recent advances and the reality behind themcited by 2
  6. 2024Interactive Explainable Anomaly Detection for Industrial Settingscited by 1
  7. 2022Anomaly Detection via Multi-Scale Contrasted Memorycited by 0
  8. 2023LBL: Logarithmic Barrier Loss Function for One-class Classificationcited by 0
  9. 2023Don't Miss Out on Novelty: Importance of Novel Features for Deep Anomaly Detectioncited by 0
  10. 2024Two Is Better Than One: Aligned Representation Pairs for Anomaly Detectioncited by 0
  11. 2024Redefining Normal: A Novel Object-Level Approach for Multi-Object Novelty Detectioncited by 0
  12. 2025Generalist Multi-Class Anomaly Detection via Distillation to Two Heterogeneous Student Networkscited by 0
  13. 2026Contrastive Knowledge Distillation for Anomaly Detection in Multi-Illumination/Focus Display Imagescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Advancing Pre-trained Teacher: Towards Robust Feature Discrepancy for Anomaly Detection
  2. 2021Student-Teacher Feature Pyramid Matching for Anomaly Detection
  3. 2024Structural Teacher-Student Normality Learning for Multi-Class Anomaly Detection and Localization
  4. 2024Dual-Modeling Decouple Distillation for Unsupervised Anomaly Detection
  5. 2025ResAD++: Towards Class Agnostic Anomaly Detection via Residual Feature Learning
  6. 2022Anomaly Detection via Reverse Distillation from One-Class Embedding
  7. 2024Dual-Student Knowledge Distillation Networks for Unsupervised Anomaly Detection
  8. 2023MixedTeacher : Knowledge Distillation for fast inference textural anomaly detection