mlsanityData to 14 Oct 2026

Shift Invariance Can Reduce Adversarial Robustness

arXiv 2103.02695 ↗First posted 3 Mar 2021cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Shift invariance is a critical property of CNNs that improves performance on classification. However, we show that invariance to circular shifts can also lead to greater sensitivity to adversarial attacks. We first characterize the margin between classes when a shift-invariant linear classifier is used. We show that the margin can only depend on the DC component of the signals. Then, using results about infinitely wide networks, we show that in some simple cases, fully connected and shift-invariant neural networks produce linear decision boundaries. Using this, we prove that shift invariance in neural networks produces adversarial examples for the simple case of two classes, each consisting of a single image with a black or white dot on a gray background. This is more than a curiosity; we show empirically that with real datasets and realistic architectures, shift invariance reduces adversarial robustness. Finally, we describe initial experiments using synthetic data to probe the source of this connection.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 49 of its 73 references that are in the map. First posted on arXiv 3 Mar 2021.

Builds on

49 of 73 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2016Densely Connected Convolutional Networkscited by 6,720
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  6. 2013Intriguing properties of neural networkscited by 5,155
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  8. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  10. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  11. 2018On the Spectral Bias of Neural Networkscited by 1,341
  12. 2016Group Equivariant Convolutional Networkscited by 1,165
37 more
  1. 2018Robustness May Be at Odds with Accuracycited by 880
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  3. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  4. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  5. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  6. 2019Making Convolutional Networks Shift-Invariant Againcited by 391
  7. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 351
  8. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  9. 2018Adversarially Robust Generalization Requires More Datacited by 318
  10. 2019High Frequency Component Helps Explain the Generalization of Convolutional Neural Networkscited by 310
  11. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  12. 2018Why do deep convolutional networks generalize so poorly to small image transformations?cited by 198
  13. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  14. 2019Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernelscited by 161
  15. 2019Making an Invisibility Cloak: Real World Adversarial Attacks on Object Detectorscited by 155
  16. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 143
  17. 2020On Translation Invariance in CNNs: Convolutional Layers can Exploit Absolute Spatial Locationcited by 142
  18. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  19. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 135
  20. 2018Are adversarial examples inevitable?cited by 113
  21. 2018Decoupling Direction and Norm for Efficient Gradient-Based L2 Adversarial Attacks and Defensescited by 109
  22. 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
  23. 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
  24. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  25. 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
  26. 2020The Recurrent Neural Tangent Kernelcited by 67
  27. 2018Excessive Invariance Causes Adversarial Vulnerabilitycited by 66
  28. 2020Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspectivecited by 60
  29. 2020Truly shift-invariant convolutional neural networkscited by 53
  30. 2020Towards Learning Convolutions from Scratchcited by 46
  31. 2018On the Geometry of Adversarial Examplescited by 43
  32. 2020Mind the Pad -- CNNs can Develop Blind Spotscited by 41
  33. 2019Adversarial Robustness May Be at Odds With Simplicitycited by 39
  34. 2019A Simple Explanation for the Existence of Adversarial Examples with Small Hamming Distancecited by 32
  35. 2019Batch Normalization is a Cause of Adversarial Vulnerabilitycited by 29
  36. 2020More Data Can Expand the Generalization Gap Between Adversarially Robust and Standard Modelscited by 27
  37. 2020Most ReLU Networks Suffer from ℓ2 Adversarial Perturbationscited by 12

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 5; 2023: 7; 2024: 2; 2025: 1.

16 papers.

Sort:

By area:

  1. 2021Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNscited by 57
  2. 2022Gradient Methods Provably Converge to Non-Robust Networkscited by 27
  3. 2023The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networkscited by 23
  4. 2022On the Limitations of Stochastic Pre-processing Defensescited by 21
  5. 2023Alias-Free Convnets: Fractional Shift Invariance via Polynomial Activationscited by 11
  6. 2022In What Ways Are Deep Neural Networks Invariant and How Should We Measure This?cited by 9
  7. 2023It Is All About Data: A Survey on the Effects of Data on Adversarial Robustnesscited by 9
  8. 2023Adversarial Examples Exist in Two-Layer ReLU Networks for Low Dimensional Linear Subspacescited by 8
  9. 2022Invariance-Aware Randomized Smoothing Certificatescited by 7
  10. 2024Improving Shift Invariance in Convolutional Neural Networks with Translation Invariant Polyphase Samplingcited by 6
  11. 2023An unsupervised approach towards promptable defect segmentation in laser-based additive manufacturing by Segment Anythingcited by 1
  12. 2024Understanding the Role of Invariance in Transfer Learningcited by 1
4 more
  1. 2025Defending Against Frequency-Based Attacks with Diffusion Modelscited by 1
  2. 2022Mitigating Overfitting in Graph Neural Networks via Feature and Hyperplane Perturbationcited by 0
  3. 2023Unveiling Invariances via Neural Network Pruningcited by 0
  4. 2023Training Image Derivatives: Increased Accuracy and Universal Robustnesscited by 0