mlsanityData to 7 Oct 2026

Comparing Spectral Bias and Robustness For Two-Layer Neural Networks: SGD vs Adaptive Random Fourier Features

arXiv 2402.00332 ↗First posted 1 Feb 2024cs.LG stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

We present experimental results highlighting two key differences resulting from the choice of training algorithm for two-layer neural networks. The spectral bias of neural networks is well known, while the spectral bias dependence on the choice of training algorithm is less studied. Our experiments demonstrate that an adaptive random Fourier features algorithm (ARFF) can yield a spectral bias closer to zero compared to the stochastic gradient descent optimizer (SGD). Additionally, we train two identically structured classifiers, employing SGD and ARFF, to the same accuracy levels and empirically assess their robustness against adversarial noise attacks.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 18 of its 34 references that are in the map. First posted on arXiv 1 Feb 2024.

Builds on

18 of 34 references.

References · 18

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2013Intriguing properties of neural networkscited by 5,114
  3. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  4. 2016Deep Learning with Differential Privacycited by 2,682
  5. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  6. 2018On the Spectral Bias of Neural Networkscited by 1,321
  7. 2016An overview of gradient descent optimization algorithmscited by 1,021
  8. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 682
  9. 2018Detecting Backdoor Attacks on Deep Neural Networks by Activation Clusteringcited by 429
  10. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
  11. 2018Training behavior of deep neural network in frequency domaincited by 174
  12. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 173
6 more
  1. 2016Generalization Properties of Learning with Random Featurescited by 163
  2. 2019A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamicscited by 49
  3. 2022On the Activation Function Dependence of the Spectral Bias of Neural Networkscited by 25
  4. 2022Friendly Noise against Adversarial Noise: A Powerful Defense against Data Poisoning Attackscited by 20
  5. 2022Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Studycited by 7
  6. 2021Wind Field Reconstruction with Adaptive Random Fourier Featurescited by 3

Built on by

Cited by year

Citing papers per year: 2024: 3.

3 papers.

  1. 2024Learnable Activation Functions in Physics-Informed Neural Networks for Solving Partial Differential Equationscited by 7
  2. 2024Adaptive Random Fourier Features Training Stabilized By Resampling With Applications in Image Regressioncited by 2
  3. 2024Deep Learning without Global Optimization by Random Fourier Neural Networkscited by 1