mlsanityData to 7 Oct 2026

Depth and Feature Learning are Provably Beneficial for Neural Network Discriminators

arXiv 2112.13867 ↗First posted 27 Dec 2021cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We construct pairs of distributions μd, nud on ℝd such that the quantity |mathbbEx ~ μd [F(x)] - mathbbEx ~ nud [F(x)]| decreases as Omega(1/d2) for some three-layer ReLU network F with polynomial width and weights, while declining exponentially in d if F is any two-layer network with polynomial weights. This shows that deep GAN discriminators are able to distinguish distributions that shallow discriminators cannot. Analogously, we build pairs of distributions μd, nud on ℝd such that |mathbbEx ~ μd [F(x)] - mathbbEx ~ nud [F(x)]| decreases as Omega(1/(dlog d)) for two-layer ReLU networks with polynomial weights, while declining exponentially for bounded-norm functions in the associated RKHS. This confirms that feature learning is beneficial for discriminators. Our bounds are based on Fourier transforms.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 28 references that are in the map. First posted on arXiv 27 Dec 2021.

Builds on

18 of 28 references.

References · 18

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  3. 2017Wasserstein GANcited by 635
  4. 2017Exploring Generalization in Deep Learningcited by 595
  5. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  6. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  7. 2015Generative Moment Matching Networkscited by 252
  8. 2017MMD GAN: Towards Deeper Understanding of Moment Matching Networkcited by 251
  9. 2016Benefits of depth in neural networkscited by 248
  10. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  11. 2015The Power of Depth for Feedforward Neural Networkscited by 181
  12. 2015Training generative neural networks via Maximum Mean Discrepancy optimizationcited by 149
6 more
  1. 2016Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networkscited by 58
  2. 2017Depth Separation for Neural Networkscited by 46
  3. 2020On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamicscited by 39
  4. 2021Depth separation beyond radial functionscited by 18
  5. 2017Learning Loss Functions for Semi-supervised Learning via Discriminative Adversarial Networkscited by 3
  6. 2021Separation Results between Fixed-Kernel and Feature-Learning Probability Metricscited by 1

Built on by

Not cited yet.