Generalization Bounds for Stochastic Gradient Langevin Dynamics: A Unified View via Information Leakage Analysis
Not on the map. Why?
Abstract (arXiv)
Recently, generalization bounds of the non-convex empirical risk minimization paradigm using Stochastic Gradient Langevin Dynamics (SGLD) have been extensively studied. Several theoretical frameworks have been presented to study this problem from different perspectives, such as information theory and stability. In this paper, we present a unified view from privacy leakage analysis to investigate the generalization bounds of SGLD, along with a theoretical framework for re-deriving previous results in a succinct manner. Aside from theoretical findings, we conduct various numerical studies to empirically assess the information leakage issue of SGLD. Additionally, our theoretical and empirical results provide explanations for prior works that study the membership privacy of SGLD.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 28 of its 45 references that are in the map. First posted on arXiv 14 Dec 2021.
Builds on
28 of 45 references.
References · 28
- 2016Deep Learning with Differential Privacycited by 2,682
- 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
- 2016Understanding deep learning requires rethinking generalizationcited by 1,657
- 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,323
- 2017Renyi Differential Privacycited by 643
- 2019On Variational Bounds of Mutual Informationcited by 550
- 2018Exploiting Unintended Feature Leakage in Collaborative Learningcited by 509
- 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
- 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
- 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
- 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
- 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 199
16 more
- 2017Generalization in Deep Learningcited by 146
- 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 118
- 2019Tightening Mutual Information Based Bounds on Generalization Errorcited by 117
- 2014Preserving Statistical Validity in Adaptive Data Analysiscited by 102
- 2015Algorithmic Stability for Adaptive Data Analysiscited by 88
- 2017Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpointscited by 82
- 2015Privacy for Free: Posterior Sampling and Stochastic Gradient Monte Carlocited by 61
- 2019On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learningcited by 57
- 2019A New Analysis of Differential Privacy's Generalization Guaranteescited by 36
- 2016On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integratorscited by 35
- 2014Preventing False Discovery in Interactive Data Analysis is Hardcited by 34
- 2019Generalization in Generative Adversarial Networks: A Novel Perspective from Privacy Protectioncited by 19
- 2015Learning with Differential Privacy: Stability, Learnability and the Sufficiency and Necessity of ERM Principlecited by 16
- 2019P3SGD: Patient Privacy Preserving SGD for Regularizing Deep CNNs in Pathological Image Classificationcited by 14
- 2016Scalable Bayesian Learning of Recurrent Neural Networks for Language Modelingcited by 9
- 2019Characterizing Membership Privacy in Stochastic Gradient Langevin Dynamicscited by 8
Built on by
Citing papers per year: 2022: 1.
1 paper.