# Transformaly -- Two (Feature Spaces) Are Better Than One

arXiv: https://arxiv.org/abs/2112.04185 · HTML: https://mlsanity.com/paper/2112.04185/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 29 of its 49 references that are in the map.
- First posted on arXiv 8 Dec 2021.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 29 of 29)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [CutPaste: Self-Supervised Learning for Anomaly Detection and Localization](https://mlsanity.com/paper/2104.04015/) (2021)
- [Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1906.12340/) (2019)
- [CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances](https://mlsanity.com/paper/2007.08176/) (2020)
- [Uninformed Students: Student-Teacher Anomaly Detection with Discriminative Latent Embeddings](https://mlsanity.com/paper/1911.02357/) (2019)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Pretrained Transformers Improve Out-of-Distribution Robustness](https://mlsanity.com/paper/2004.06100/) (2020)
- [Deep Anomaly Detection Using Geometric Transformations](https://mlsanity.com/paper/1805.10917/) (2018)
- [VT-ADL: A Vision Transformer Network for Image Anomaly Detection and Localization](https://mlsanity.com/paper/2104.10036/) (2021)
- [Classification-Based Anomaly Detection for General Data](https://mlsanity.com/paper/2005.02359/) (2020)
- [PANDA: Adapting Pretrained Features for Anomaly Detection and Segmentation](https://mlsanity.com/paper/2010.05903/) (2020)
- [Learning and Evaluating Representations for Deep One-class Classification](https://mlsanity.com/paper/2011.02578/) (2020)
- [Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifiers](https://mlsanity.com/paper/1809.03576/) (2018)
- [Efficient GAN-Based Anomaly Detection](https://mlsanity.com/paper/1802.06222/) (2018)
- [Deep Nearest Neighbor Anomaly Detection](https://mlsanity.com/paper/2002.10445/) (2020)
- [Inpainting Transformer for Anomaly Detection](https://mlsanity.com/paper/2104.13897/) (2021)
- [Mean-Shifted Contrastive Loss for Anomaly Detection](https://mlsanity.com/paper/2106.03844/) (2021)
- [Anomaly Detection with Generative Adversarial Networks for Multivariate Time Series](https://mlsanity.com/paper/1809.04758/) (2018)
- [Detecting semantic anomalies](https://mlsanity.com/paper/1908.04388/) (2019)
- [Puzzle-AE: Novelty Detection in Images through Solving Puzzles](https://mlsanity.com/paper/2008.12959/) (2020)
- [Do We Really Need to Learn Representations from In-domain Data for Outlier Detection?](https://mlsanity.com/paper/2105.09270/) (2021)

## Built on by (most cited first, 25 of 25)

- [Improving neural network representations using human similarity judgments](https://mlsanity.com/paper/2306.04507/) (2023)
- [GeneralAD: Anomaly Detection Across Domains by Attending to Distorted Features](https://mlsanity.com/paper/2407.12427/) (2024)
- [Set Features for Fine-grained Anomaly Detection](https://mlsanity.com/paper/2302.12245/) (2023)
- [PatchGuard: Adversarially Robust Anomaly Detection and Localization through Vision Transformers and Pseudo Anomalies](https://mlsanity.com/paper/2506.09237/) (2025)
- [Foundation Models and Transformers for Anomaly Detection: A Survey](https://mlsanity.com/paper/2507.15905/) (2025)
- [Unveiling the Unseen: A Comprehensive Survey on Explainable Anomaly Detection in Images and Videos](https://mlsanity.com/paper/2302.06670/) (2023)
- [No Free Lunch: The Hazards of Over-Expressive Representations in Anomaly Detection](https://mlsanity.com/paper/2306.07284/) (2023)
- [A graph neural network-based model with Out-of-Distribution Robustness for enhancing Antiretroviral Therapy Outcome Prediction for HIV-1](https://mlsanity.com/paper/2312.17506/) (2023)
- [RODEO: Robust Outlier Detection via Exposing Adaptive Out-of-Distribution Samples](https://mlsanity.com/paper/2501.16971/) (2025)
- [Adversarially Robust Out-of-Distribution Detection Using Lyapunov-Stabilized Embeddings](https://mlsanity.com/paper/2410.10744/) (2024)
- [Fake It Till You Make It: Towards Accurate Near-Distribution Novelty Detection](https://mlsanity.com/paper/2205.14297/) (2022)
- [Killing it with Zero-Shot: Adversarially Robust Novelty Detection](https://mlsanity.com/paper/2501.15271/) (2025)
- [Positive Difference Distribution for Image Outlier Detection using Normalizing Flows and Contrastive Data](https://mlsanity.com/paper/2208.14024/) (2022)
- [Dissolving Is Amplifying: Towards Fine-Grained Anomaly Detection](https://mlsanity.com/paper/2302.14696/) (2023)
- [Set Features for Anomaly Detection](https://mlsanity.com/paper/2311.14773/) (2023)
- [Understanding normalization in contrastive representation learning and out-of-distribution detection](https://mlsanity.com/paper/2312.15288/) (2023)
- [Critical Review for One-class Classification: recent advances and the reality behind them](https://mlsanity.com/paper/2404.17931/) (2024)
- [Interactive Explainable Anomaly Detection for Industrial Settings](https://mlsanity.com/paper/2410.12817/) (2024)
- [Anomaly Detection via Multi-Scale Contrasted Memory](https://mlsanity.com/paper/2211.09041/) (2022)
- [LBL: Logarithmic Barrier Loss Function for One-class Classification](https://mlsanity.com/paper/2307.10753/) (2023)
- [Don't Miss Out on Novelty: Importance of Novel Features for Deep Anomaly Detection](https://mlsanity.com/paper/2310.00797/) (2023)
- [Two Is Better Than One: Aligned Representation Pairs for Anomaly Detection](https://mlsanity.com/paper/2405.18848/) (2024)
- [Redefining Normal: A Novel Object-Level Approach for Multi-Object Novelty Detection](https://mlsanity.com/paper/2412.11148/) (2024)
- [Generalist Multi-Class Anomaly Detection via Distillation to Two Heterogeneous Student Networks](https://mlsanity.com/paper/2509.24448/) (2025)
- [Contrastive Knowledge Distillation for Anomaly Detection in Multi-Illumination/Focus Display Images](https://mlsanity.com/paper/2609.05520/) (2026)

## Abstract (arXiv)

Anomaly detection is a well-established research area that seeks to identify samples outside of a predetermined distribution. An anomaly detection pipeline is comprised of two main stages: (1) feature extraction and (2) normality score assignment. Recent papers used pre-trained networks for feature extraction achieving state-of-the-art results. However, the use of pre-trained networks does not fully-utilize the normal samples that are available at train time. This paper suggests taking advantage of this information by using teacher-student training. In our setting, a pretrained teacher network is used to train a student network on the normal training samples. Since the student network is trained only on normal samples, it is expected to deviate from the teacher network in abnormal cases. This difference can serve as a complementary representation to the pre-trained feature vector. Our method -- Transformaly -- exploits a pre-trained Vision Transformer (ViT) to extract both feature vectors: the pre-trained (agnostic) features and the teacher-student (fine-tuned) features. We report state-of-the-art AUROC results in both the common unimodal setting, where one class is considered normal and the rest are considered abnormal, and the multimodal setting, where all classes but one are considered normal, and just one class is considered abnormal. The code is available at https://github.com/MatanCohen1/Transformaly.