mlsanityData to 14 Oct 2026

What Variables Affect Out-of-Distribution Generalization in Pretrained Models?

arXiv 2405.15018 ↗First posted 23 May 2024cs.LG cs.AI cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

Embeddings produced by pre-trained deep neural networks (DNNs) are widely used; however, their efficacy for downstream tasks can vary widely. We study the factors influencing transferability and out-of-distribution (OOD) generalization of pre-trained DNN embeddings through the lens of the tunnel effect hypothesis, which is closely related to intermediate neural collapse. This hypothesis suggests that deeper DNN layers compress representations and hinder OOD generalization. Contrary to earlier work, our experiments show this is not a universal phenomenon. We comprehensively investigate the impact of DNN architecture, training data, image resolution, and augmentations on transferability. We identify that training with high-resolution datasets containing many classes greatly reduces representation compression and improves transferability. Our results emphasize the danger of generalizing findings from toy datasets to broader contexts.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 81 of its 101 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

81 of 101 references.

References · 81

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  11. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
38 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  2. 2022A ConvNet for the 2020scited by 3,579
  3. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  4. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  6. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  7. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  8. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  9. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  10. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  11. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  12. 2019Contrastive Multiview Codingcited by 1,301
  13. 2014How transferable are features in deep neural networks?cited by 1,225
  14. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  15. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  16. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  17. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  18. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 712
  19. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  20. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
  21. 2018Do Better ImageNet Models Transfer Better?cited by 565
  22. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 454
  23. 2018Generalizing to Unseen Domains via Adversarial Data Augmentationcited by 451
  24. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 424
  25. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  26. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  27. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  28. 2019Towards Efficient Data Valuation Based on the Shapley Valuecited by 281
  29. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  30. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 249
  31. 2021MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Spacecited by 233
  32. 2019REMIND Your Neural Network to Prevent Catastrophic Forgettingcited by 197
  33. 2019Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathologycited by 173
  34. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 161
  35. 2023Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspectivecited by 138
  36. 2023In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluationcited by 134
  37. 2020What makes instance discrimination good for transfer learning?cited by 111
  38. 2020On Robustness and Transferability of Convolutional Neural Networkscited by 98

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 7.

13 papers.

Sort:

By area:

  1. 2025Controlling Neural Collapse Enhances Out-of-Distribution Detection and Transfer Learningcited by 11
  2. 2025Unpacking Softmax: How Temperature Drives Representation Collapse, Compression, and Generalizationcited by 3
  3. 2026Diagnosing Generalization Failures from Representational Geometry Markerscited by 2
  4. 2025A Good Start Matters: Enhancing Continual Learning with Data-Driven Weight Initializationcited by 1
  5. 2025Is Supervised Learning Really That Different from Unsupervised?cited by 0
  6. 2025What is the role of memorization in Continual Learning?cited by 0
  7. 2025Neural Coherence : Find higher performance to out-of-distribution tasks from few samplescited by 0
  8. 2026Generalizable Prompt Tuning for Audio-Language Models via Semantic Expansioncited by 0
  9. 2026Unlocking ImageNet's Multi-Object Nature: Automated Large-Scale Multilabel Annotationcited by 0
  10. 2026Ranked Activation Shift for Post-Hoc Out-of-Distribution Detectioncited by 0
  11. 2026Sparsity, Superposition, and Forgetting: A Mechanistic Study of Representation Retention in Continual Learningcited by 0
  12. 2026Lifelong Representations: A Survey on Continual Self-Supervised Learning for Vision Modelscited by 0
1 more
  1. 2026Topological Attribution Distance (TAD): Revealing Segment-Level RAG Influence on LLM Output Geometry for Incident Log Analysiscited by 0