mlsanityData to 7 Oct 2026

Deep Regression Ensembles

arXiv 2203.05417 ↗First posted 10 Mar 2022stat.ML cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

We introduce a methodology for designing and training deep neural networks (DNN) that we call "Deep Regression Ensembles" (DRE). It bridges the gap between DNN and two-layer neural networks trained with random feature regression. Each layer of DRE has two components, randomly drawn input weights and output weights trained myopically (as if the final output layer) using linear ridge regression. Within a layer, each neuron uses a different subset of inputs and a different ridge penalty, constituting an ensemble of random feature ridge regressions. Our experiments show that a single DRE architecture is at par with or exceeds state-of-the-art DNN in many data sets. Yet, because DRE neural weights are either known in closed-form or randomly drawn, its computational cost is orders of magnitude smaller than DNN.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 44 of its 59 references that are in the map. First posted on arXiv 10 Mar 2022.

Builds on

44 of 59 references.

References · 44

  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2018On Lazy Training in Differentiable Programmingcited by 671
  7. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  8. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  9. 2017Deep Neural Networks as Gaussian Processescited by 617
  10. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  11. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  12. 2019Benign Overfitting in Linear Regressioncited by 544
32 more
  1. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  2. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  3. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  4. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  5. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  6. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  7. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  8. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  9. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  10. 2019Linearized two-layers neural networks in high dimensioncited by 155
  11. 2020Benign overfitting in ridge regressioncited by 152
  12. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  13. 2014Difference Target Propagationcited by 144
  14. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  15. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
  16. 2019Training Neural Networks with Local Error Signalscited by 135
  17. 2018Greedy Layerwise Learning Can Scale to ImageNetcited by 113
  18. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  19. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  20. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  21. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  22. 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
  23. 2019The HSIC Bottleneck: Deep Learning without Back-Propagationcited by 77
  24. 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
  25. 2020A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descentcited by 69
  26. 2016Training Neural Networks Without Gradients: A Scalable ADMM Approachcited by 61
  27. 2020Neural Kernels Without Tangentscited by 56
  28. 2018Beyond Backprop: Online Alternating Minimization with Auxiliary Variablescited by 23
  29. 2019A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learningcited by 20
  30. 2018The Dynamics of Learning: A Random Matrix Approachcited by 15
  31. 2019The capacity of feedforward neural networkscited by 14
  32. 2019Disentangling feature and lazy training in deep neural networkscited by 10

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023Deep incremental learning models for financial temporal tabular datasets with distribution shiftscited by 1