mlsanityData to 14 Oct 2026

Invisible Shortcuts: Why Vision Encoders Know Your Camera

arXiv 2608.05424 ↗First posted 5 Aug 2026cs.CV cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Deep vision models exploit shortcuts, relying on cues that correlate with supervision signals. Prior work has focused on visible biases, such as object-background or texture correlations. We identify a different source of shortcut learning: invisible metadata traces embedded at the pixel level, for metadata such as image processing and photo acquisition. We hypothesize that large-scale semantic supervision, whether through categorical labels (ImageNet) or billion-scale captions (LAION), naturally induces metadata-semantics correlations during pretraining, leading models to convert low-level signals into predictive features. By introducing controlled metadata-semantics correlations, we show that stronger ones produce systematically higher sensitivity to metadata traces and larger performance degradation under metadata distribution shifts. We further explore mitigation strategies applied during and after pretraining that reduce sensitivity not only to targeted metadata but also to unseen ones, without sacrificing performance on downstream tasks. Metadata sensitivity also has a positive side: it partly explains the strong generated-image detection ability of some encoders, while its mitigation can improve out-of-distribution generalization. Code: https://github.com/ryan-caesar-ramos/visual-encoder-traces

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 77 of its 84 references that are in the map. First posted on arXiv 5 Aug 2026.

Builds on

77 of 84 references.

References · 77

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  10. 2022A ConvNet for the 2020scited by 3,579
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  12. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
38 more
  1. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  2. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  3. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  4. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  5. 2019Invariant Risk Minimizationcited by 1,854
  6. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  7. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  8. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  9. 2019Natural Adversarial Examplescited by 1,408
  10. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  11. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  12. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  13. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 801
  14. 2018Recognition in Terra Incognitacited by 658
  15. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 523
  16. 2023Towards Universal Fake Image Detectors that Generalize Across Generative Modelscited by 490
  17. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 381
  18. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 379
  19. 2021ResNet strikes back: An improved training procedure in timmcited by 368
  20. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 351
  21. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  22. 2018Learning Not to Learn: Training Deep Neural Networks with Biased Datacited by 255
  23. 2019On the "steerability" of generative adversarial networkscited by 248
  24. 2022Improving Out-of-Distribution Robustness via Selective Augmentationcited by 202
  25. 2019Learning De-biased Representations with Biased Representationscited by 199
  26. 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 172
  27. 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clonescited by 172
  28. 2022DE-FAKE: Detection and Attribution of Fake Images Generated by Text-to-Image Generation Modelscited by 168
  29. 2019REPAIR: Removing Representation Bias by Dataset Resamplingcited by 157
  30. 2022FlexiViT: One Model for All Patch Sizescited by 152
  31. 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
  32. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  33. 2020Learning from Failure: Training Debiased Classifier from Biased Classifiercited by 133
  34. 2019Learning Robust Representations by Projecting Superficial Statistics Outcited by 123
  35. 2018Turning a Blind Eye: Explicit Removal of Biases and Variation from Deep Neural Network Embeddingscited by 114
  36. 2021Salient ImageNet: How to discover spurious features in Deep Learning?cited by 114
  37. 2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 108
  38. 2022Spread Spurious Attribute: Improving Worst-group Accuracy with Spurious Attribute Estimationcited by 88

50 most cited of 77 All 77 (JSON)

Built on by

Not cited yet.