mlsanityData to 7 Oct 2026

Testing predictions of representation cost theory with CNNs

arXiv 2210.01257 ↗First posted 3 Oct 2022cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

It is widely acknowledged that trained convolutional neural networks (CNNs) have different levels of sensitivity to signals of different frequency. In particular, a number of empirical studies have documented CNNs sensitivity to low-frequency signals. In this work we show with theory and experiments that this observed sensitivity is a consequence of the frequency distribution of natural images, which is known to have most of its power concentrated in low-to-mid frequencies. Our theoretical analysis relies on representations of the layers of a CNN in frequency space, an idea that has previously been used to accelerate computations and study implicit bias of network training algorithms, but to the best of our knowledge has not been applied in the domain of model robustness.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 50 references that are in the map. First posted on arXiv 3 Oct 2022.

Builds on

24 of 50 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  4. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  5. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  6. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  7. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  8. 2019A Fourier Perspective on Model Robustness in Computer Visioncited by 288
  9. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  10. 2021Learning to See by Looking at Noisecited by 92
  11. 2013Fast Training of Convolutional Networks through FFTscited by 89
  12. 2018Low Frequency Adversarial Perturbationcited by 80
12 more
  1. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 74
  2. 2021A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustnesscited by 62
  3. 2019On the Effectiveness of Low Frequency Perturbationscited by 56
  4. 2016Wav2Letter: an End-to-End ConvNet-based Speech Recognition Systemcited by 43
  5. 2021A Frequency Perspective of Adversarial Robustnesscited by 28
  6. 2014Fast Convolutional Nets With fbfft: A GPU Performance Evaluationcited by 27
  7. 2022projUNN: efficient method for training deep networks with unitary matricescited by 27
  8. 2022Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regressioncited by 22
  9. 2021Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Normcited by 18
  10. 2021Implicit Bias of Linear Equivariant Networkscited by 18
  11. 2021Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networkscited by 17
  12. 2021Impact of Spatial Frequency Based Constraints on Adversarial Robustnesscited by 8

Built on by

Not cited yet.