mlsanityData to 7 Oct 2026

Minimax optimality of deep neural networks on dependent data via PAC-Bayes bounds

arXiv 2410.21702 ↗First posted 29 Oct 2024stat.ML cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

In a groundbreaking work, Schmidt-Hieber (2020) proved the minimax optimality of deep neural networks with ReLu activation for least-square regression estimation over a large class of functions defined by composition. In this paper, we extend these results in many directions. First, we remove the i.i.d. assumption on the observations, to allow some time dependence. The observations are assumed to be a Markov chain with a non-null pseudo-spectral gap. Then, we study a more general class of machine learning problems, which includes least-square and logistic regression as special cases. Leveraging on PAC-Bayes oracle inequalities and a version of Bernstein inequality due to Paulin (2015), we derive upper bounds on the estimation risk for a generalized Bayesian estimator. In the case of least-square regression, this bound matches (up to a logarithmic factor) the lower bound of Schmidt-Hieber (2020). We establish a similar lower bound for classification with the logistic loss, and prove that the proposed DNN estimator is optimal in the minimax sense.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 31 of its 68 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

31 of 68 references.

References · 31

  1. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  2. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  3. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  4. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  5. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
  6. 2019Practical Deep Learning with Bayesian Principlescited by 158
  7. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  8. 2019Deep ReLU network approximation of functions on a manifoldcited by 73
  9. 2023Generalization Bounds: Perspectives from Information Theory and PAC-Bayescited by 60
  10. 2018Posterior Concentration for Sparse Deep Learningcited by 50
  11. 2021Data augmentation in Bayesian neural networks and the cold posterior effectcited by 36
  12. 2019Smooth function approximation by deep neural networks with general activation functionscited by 29
19 more
  1. 2019Convergence Rates of Variational Inference in Sparse Deep Learningcited by 27
  2. 2021Consistent Sparse Deep Learning: Theory and Computationcited by 27
  3. 2017Mixing time estimation in reversible Markov chains from a single sample pathcited by 26
  4. 2021Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effectcited by 22
  5. 2020Nonconvex sparse regularization for deep neural networks and its optimalitycited by 21
  6. 2022Theoretical analysis of deep neural networks for temporally dependent observationscited by 16
  7. 2023Classification with Deep Neural Networks and Logistic Losscited by 16
  8. 2020On the rate of convergence of a deep recurrent neural network estimate in a regression problem with dependent datacited by 14
  9. 2022Adaptive deep learning for nonlinear time series modelscited by 11
  10. 2024FSP-Laplace: Function-Space Priors for the Laplace Approximation in Bayesian Deep Learningcited by 11
  11. 2021Wide stochastic networks: Gaussian limit and PAC-Bayesian trainingcited by 10
  12. 2023A reduced-rank approach to predicting multiple binary responses through machine learningcited by 10
  13. 2023Excess risk bound for deep learning under weak dependencecited by 9
  14. 2024Posterior and variational inference for deep neural networks with heavy-tailed weightscited by 9
  15. 2023Sparse-penalized deep neural networks estimator under weak dependencecited by 8
  16. 2024Posterior concentrations of fully-connected Bayesian neural networks with general priors on the weightscited by 7
  17. 2024Misclassification bounds for PAC-Bayesian sparse deep learningcited by 7
  18. 2024Deep Horseshoe Gaussian Processescited by 5
  19. 2023Penalized deep neural networks estimator with general loss functions under weak dependencecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Empirical PAC-Bayes Bounds for Markov Chainscited by 2
  2. 2025A general framework for deep learningcited by 2
  3. 2026Deep regression learning from dependent observations with minimum error entropy principlecited by 0
  4. 2026Adaptive deep nonparametric regression from dependent data under covariate shiftcited by 0