mlsanityData to 7 Oct 2026

Differential Privacy Mechanisms in Neural Tangent Kernel Regression

arXiv 2407.13621 ↗First posted 18 Jul 2024cs.LG cs.AI cs.CRcited by 14

Not on the map. Why?

Abstract (arXiv)

Training data privacy is a fundamental problem in modern Artificial Intelligence (AI) applications, such as face recognition, recommendation systems, language generation, and many others, as it may contain sensitive user information related to legal issues. To fundamentally understand how privacy mechanisms work in AI applications, we study differential privacy (DP) in the Neural Tangent Kernel (NTK) regression setting, where DP is one of the most powerful tools for measuring privacy under statistical learning, and NTK is one of the most popular analysis frameworks for studying the learning mechanisms of deep neural networks. In our work, we can show provable guarantees for both differential privacy and test accuracy of our NTK regression. Furthermore, we conduct experiments on the basic image classification dataset CIFAR10 to demonstrate that NTK regression can preserve good accuracy under a modest privacy budget, supporting the validity of our analysis. To our knowledge, this is the first work to provide a DP guarantee for NTK regression.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 105 of its 142 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

105 of 142 references.

References · 105

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2016Deep Learning with Differential Privacycited by 2,682
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  8. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  9. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  10. 2018On Lazy Training in Differentiable Programmingcited by 671
  11. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  12. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
38 more
  1. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  2. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  3. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  4. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  5. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  6. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  7. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  8. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  9. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  10. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  11. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  12. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  13. 2021Numerical Composition of Differential Privacycited by 174
  14. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 173
  15. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  16. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  17. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  18. 2017Deep Learning based Recommender System: A Survey and New Perspectivescited by 136
  19. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  20. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 120
  21. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 118
  22. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  23. 2018Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?cited by 87
  24. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  25. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  26. 2018On the Convergence Rate of Training Recurrent Neural Networkscited by 82
  27. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  28. 2022Differentially Private Language Models for Secure Data Sharingcited by 68
  29. 2019Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobiancited by 67
  30. 2020When Machine Learning Meets Privacy: A Survey and Outlookcited by 67
  31. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  32. 2021Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Timecited by 62
  33. 2019Quadratic Suffices for Over-parametrization via Matrix Chernoff Boundcited by 58
  34. 2020Training (Overparametrized) Neural Networks in Near-Linear Timecited by 58
  35. 2022Mixed Differential Privacy in Computer Visioncited by 53
  36. 2022A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Featurescited by 49
  37. 2022When Does Differentially Private Learning Not Suffer in High Dimensions?cited by 48
  38. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48

50 most cited of 105 All 105 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 9.

14 papers.

Sort:

By area:

  1. 2024LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformerscited by 35
  2. 2024Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Timecited by 22
  3. 2025High-Order Matching for One-Step Shortcut Diffusion Modelscited by 15
  4. 2025Force Matching with Relativistic Constraints: A Physics-Inspired Approach to Stable and Efficient Generative Modelingcited by 12
  5. 2025Theoretical Guarantees for High Order Trajectory Refinement in Generative Flowscited by 9
  6. 2025Fast RoPE Attention: Combining the Polynomial Method and Fast Fourier Transformcited by 9
  7. 2024Fast John Ellipsoid Computation with Differential Privacy Optimizationcited by 7
  8. 2025Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapsecited by 7
  9. 2025Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel Approachescited by 4
  10. 2025HOFAR: High-Order Augmentation of Flow Autoregressive Transformerscited by 4
  11. 2024Inverting the Leverage Score Gradient: An Efficient Approximate Newton Methodcited by 3
  12. 2024Differentially Private Kernel Density Estimationcited by 2
2 more
  1. 2025Theoretical Foundation of Flow-Based Time Series Generation: Provable Approximation, Generalization, and Efficiencycited by 2
  2. 2025How Private is Your Attention? Bridging Privacy with In-Context Learningcited by 0