mlsanityData to 14 Oct 2026

What Drives the Inlier-Memorization Effect? A Theory of Outlier Detection via Early Training Dynamics

arXiv 2606.29791 ↗First posted 29 Jun 2026cs.LG cs.AI stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Outlier detection (OD) aims to identify anomalous instances by learning the underlying structure of normal data (inliers), and is particularly challenging in fully unsupervised settings where no information about anomalies is available during training. Recent advances have leveraged the inlier-memorization (IM) effect, a phenomenon in which deep models memorize inlier patterns earlier than those of outliers, as a powerful signal for distinguishing outliers. However, despite its empirical success, the theoretical understanding of the IM effect remains limited. In this work, we present a theoretical study of the IM effect. Focusing on a simple autoencoder, we show that, under mild assumptions, the model can successfully memorize inliers while failing to memorize outliers during certain stages of early training. In particular, we characterize not only the emergence of the IM effect, but also its strength and persistence, and analyze how these properties depend on the data distribution and parameter initialization. In addition, building on these insights, we derive simple yet practical guidelines for enhancing the IM effect, including data preprocessing and parameter initialization schemes, achieving state-of-the-art performance on the ADBench datasets. Our findings provide a theoretical foundation for the IM effect and offer actionable directions for improving IM-based outlier detection methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 63 references that are in the map. First posted on arXiv 29 Jun 2026.

Builds on

37 of 63 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  7. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  8. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  9. 2017Deep Neural Networks as Gaussian Processescited by 622
  10. 2017Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discoverycited by 566
  11. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  12. 2020CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instancescited by 441
25 more
  1. 2019Deep Learning for Anomaly Detection: A Surveycited by 358
  2. 2019Deep Semi-Supervised Anomaly Detectioncited by 291
  3. 2018Deep Anomaly Detection Using Geometric Transformationscited by 269
  4. 2022ADBench: Anomaly Detection Benchmarkcited by 232
  5. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 215
  6. 2020Classification-Based Anomaly Detection for General Datacited by 211
  7. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  8. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  9. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 159
  10. 2022ECOD: Unsupervised Outlier Detection Using Empirical Cumulative Distribution Functionscited by 153
  11. 2020COPOD: Copula-Based Outlier Detectioncited by 146
  12. 2020Deep Learning for Medical Anomaly Detection -- A Surveycited by 127
  13. 2020DROCC: Deep Robust One-Class Classificationcited by 121
  14. 2016Deep Structured Energy Based Models for Anomaly Detectioncited by 114
  15. 2021Neural Transformation Learning for Deep Anomaly Detection Beyond Imagescited by 99
  16. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  17. 2023On Diffusion Modeling for Anomaly Detectioncited by 72
  18. 2018Generative Adversarial Active Learning for Unsupervised Outlier Detectioncited by 65
  19. 2022Hyperparameter Sensitivity in Deep Outlier Detection: Analysis and a Scalable Hyper-Ensemble Solutioncited by 24
  20. 2019Benefits of Jointly Training Autoencoders: An Improved Neural Tangent Kernel Analysiscited by 13
  21. 2023ODIM: Outlier Detection via Likelihood of Under-Fitted Generative Modelscited by 7
  22. 2024On the Convergence Analysis of Over-Parameterized Variational Autoencoders: A Neural Tangent Kernel Perspectivecited by 3
  23. 2024GradStop: Exploring Training Dynamics in Unsupervised Outlier Detection through Gradientcited by 3
  24. 2024ALTBI: Constructing Improved Outlier Detection Models via Optimization of Inlier-Memorization Effectcited by 1
  25. 2026Memorize Early, Then Query: Inlier-Memorization-Guided Active Outlier Detectioncited by 1

Built on by

Not cited yet.