Separation Results between Fixed-Kernel and Feature-Learning Probability Metrics
Not on the map. Why?
Abstract (arXiv)
Several works in implicit and explicit generative modeling empirically observed that feature-learning discriminators outperform fixed-kernel discriminators in terms of the sample quality of the models. We provide separation results between probability metrics with fixed-kernel and feature-learning discriminators using the function classes F2 and F1 respectively, which were developed to study overparametrized two-layer neural networks. In particular, we construct pairs of distributions over hyper-spheres that can not be discriminated by fixed kernel (F2) integral probability metric (IPM) and Stein discrepancy (SD) in high dimensions, but that can be discriminated by their feature learning (F1) counterparts. To further study the separation we provide links between the F1 and F2 IPMs with sliced Wasserstein distances. Our work suggests that fixed-kernel discriminators perform worse than their feature learning counterparts because their corresponding metrics are weaker.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 34 of its 51 references that are in the map. First posted on arXiv 10 Jun 2021.
Builds on
34 of 51 references.
References · 34
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2015Variational Inference with Normalizing Flowscited by 2,161
- 2017Wasserstein GANcited by 635
- 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
- 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
- 2017Learning Generative Models with Sinkhorn Divergencescited by 310
- 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
- 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
- 2015Generative Moment Matching Networkscited by 252
- 2017MMD GAN: Towards Deeper Understanding of Moment Matching Networkcited by 251
- 2016A Kernelized Stein Discrepancy for Goodness-of-fit Tests and Model Evaluationcited by 236
- 2019Generalized Sliced Wasserstein Distancescited by 205
22 more
- 2015The Power of Depth for Feedforward Neural Networkscited by 181
- 2017Generalization and Equilibrium in Generative Adversarial Nets (GANs)cited by 165
- 2015Training generative neural networks via Maximum Mean Discrepancy optimizationcited by 149
- 2016A Kernel Test of Goodness of Fitcited by 143
- 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
- 2016Learning in Implicit Generative Modelscited by 124
- 2017Measuring Sample Quality with Kernelscited by 119
- 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 113
- 2019How do infinite width bounded norm networks look in function space?cited by 105
- 2015Measuring Sample Quality with Stein's Methodcited by 94
- 2019Subspace Robust Wasserstein Distancescited by 89
- 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
- 2020Statistical and Topological Properties of Sliced Probability Divergencescited by 83
- 2015Sliced Wasserstein Kernels for Probability Distributionscited by 64
- 2019Minimum Stein Discrepancy Estimatorscited by 57
- 2019Gradient Dynamics of Shallow Univariate ReLU Networkscited by 46
- 2020On Projection Robust Optimal Transport: Sample Complexity and Model Misspecificationcited by 34
- 2018Random Feature Stein Discrepanciescited by 23
- 2019Learning Implicit Generative Models by Matching Perceptual Featurescited by 18
- 2021Depth separation beyond radial functionscited by 18
- 2021On Energy-Based Models with Overparametrized Shallow Neural Networkscited by 8
- 2020Kernel Stein Generative Modelingcited by 5
Built on by
Citing papers per year: 2021: 1.
1 paper.