mlsanityData to 14 Oct 2026

Whitening Improves Robustness to Spurious Correlations in Linear Probes

arXiv 2609.39177 ↗First posted 30 Sep 2026cs.LGcited by 0

On the map: Mechanistic interpretability → Sparse autoencoder training and evaluation

Abstract (arXiv)

Deep neural networks tend to rely on simple features that may be spurious and thus fail to generalize. We study this problem in the setting of linear probes, where a (generalized) linear model is fitted on the representations of a (pretrained) model. We use the connection of these models to the max-margin classifier, and show they favor directions associated with large eigenvalues of the covariance matrix. Whitening removes this preference by equalizing the eigenvalues of the covariance matrix. This observation motivates whitening as a preprocessing step that can reduce reliance on spurious correlations without requiring prior knowledge of their presence or labeled data. We examine the effect of whitening on a synthetic data-generating process and standard spurious correlation benchmarks, and find that it improves robustness. We also find that whitening can improve robustness when added to existing approaches.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability. Builds on 59 of its 68 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

59 of 68 references.

References · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  3. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  5. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  6. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  7. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  8. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  9. 2018On the Spectral Bias of Neural Networkscited by 1,341
  10. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  11. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  12. 2017A Closer Look at Memorization in Deep Networkscited by 942
38 more
  1. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  2. 2018Do Better ImageNet Models Transfer Better?cited by 565
  3. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 523
  4. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 424
  5. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 381
  6. 2019Unmasking Clever Hans Predictors and Assessing What Machines Really Learncited by 365
  7. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 351
  8. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 343
  9. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  10. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  11. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  12. 2020Gradient Starvation: A Learning Proclivity in Neural Networkscited by 267
  13. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 254
  14. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  15. 2020Concept Whitening for Interpretable Image Recognitioncited by 225
  16. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
  17. 2021Simple data balancing achieves competitive worst-group-accuracycited by 178
  18. 2016Correlation Alignment for Unsupervised Domain Adaptationcited by 147
  19. 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
  20. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  21. 2018Convergence of Gradient Descent on Separable Datacited by 110
  22. 2019Iterative Normalization: Beyond Standardization towards Efficient Whiteningcited by 92
  23. 2018Decorrelated Batch Normalizationcited by 83
  24. 2021Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions with Superior OOD Generalizationcited by 83
  25. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
  26. 2023Simple and Fast Group Robustness by Automatic Feature Reweightingcited by 68
  27. 2019Unsupervised Domain Adaptation using Feature-Whitening and Consensus Losscited by 67
  28. 2020What Do Neural Networks Learn When Trained With Random Labels?cited by 59
  29. 2023Towards Last-layer Retraining for Group Robustness with Fewer Annotationscited by 54
  30. 2023Identifying Spurious Biases Early in Training through the Lens of Simplicity Biascited by 44
  31. 2024The Clever Hans Mirage: A Comprehensive Survey on Spurious Correlations in Machine Learningcited by 41
  32. 2023Don't blame Dataset Shift! Shortcut Learning due to Gradients and Cross Entropycited by 30
  33. 2022Are All Spurious Features in Natural Language Alike? An Analysis through a Causal Lenscited by 28
  34. 2023Simplicity Bias in 1-Hidden Layer Neural Networkscited by 26
  35. 2022Freeze then Train: Towards Provable Representation Learning under Spurious Correlations and Feature Noisecited by 22
  36. 2024The Pitfalls of Memorization: When Memorization Hurts Generalizationcited by 19
  37. 2020Whitening and second order optimization both make information in the dataset unusable during training, and can reduce or prevent generalizationcited by 15
  38. 2024Frozen Feature Augmentation for Few-Shot Image Classificationcited by 13

50 most cited of 59 All 59 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Data Whitening Improves Sparse Autoencoder Learning
  2. 2026Zeta: Dual Whitening for Matrix Optimization via Coordinate-Adaptive Preconditioning
  3. 2025ACE and Diverse Generalization via Selective Disagreement
  4. 2024Interpreting Global Perturbation Robustness of Image Models using Axiomatic Spectral Importance Decomposition
  5. 2022Improving Robustness by Enhancing Weak Subnets
  6. 2026Self-Distillation is Optimal Among Spectral Shrinkage Estimators in Spiked Covariance Models
  7. 2026Why Some Models Resist Unlearning: A Linear Stability Perspective
  8. 2026Feature-Space Smoothing: Certified Robustness of Deep Representations