mlsanityData to 7 Oct 2026

Multiple Descent in the Multiple Random Feature Model

arXiv 2208.09897 ↗First posted 21 Aug 2022math.ST cs.LG stat.MLcited by 10

Not on the map. Why?

Abstract (arXiv)

Recent works have demonstrated a double descent phenomenon in over-parameterized learning. Although this phenomenon has been investigated by recent works, it has not been fully understood in theory. In this paper, we investigate the multiple descent phenomenon in a class of multi-component prediction models. We first consider a ''double random feature model'' (DRFM) concatenating two types of random features, and study the excess risk achieved by the DRFM in ridge regression. We calculate the precise limit of the excess risk under the high dimensional framework where the training sample size, the dimension of data, and the dimension of random features tend to infinity proportionally. Based on the calculation, we further theoretically demonstrate that the risk curves of DRFMs can exhibit triple descent. We then provide a thorough experimental study to verify our theory. At last, we extend our study to the ''multiple random feature model'' (MRFM), and show that MRFMs ensembling K types of random features may exhibit (K+1)-fold descent. Our analysis points out that risk curves with a specific number of descent generally exist in learning multi-component prediction models.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 27 of its 42 references that are in the map. First posted on arXiv 21 Aug 2022.

Builds on

27 of 42 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  4. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  5. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  6. 2019Two models of double descent for weak featurescited by 194
  7. 2020Optimal Regularization Can Mitigate Double Descentcited by 107
  8. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  9. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  10. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  11. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  12. 2019Scaling description of generalization with number of parameters in deep learningcited by 83
15 more
  1. 2019A Model of Double Descent for High-dimensional Binary Linear Classificationcited by 81
  2. 2020Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regimecited by 81
  3. 2020Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domainscited by 78
  4. 2020Understanding Double Descent Requires a Fine-Grained Bias-Variance Decompositioncited by 75
  5. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  6. 2020A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descentcited by 69
  7. 2020Triple descent and the two kinds of overfitting: Where & why do they appear?cited by 64
  8. 2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 48
  9. 2020Multiple Descent: Design Your Own Generalization Curvecited by 44
  10. 2022Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regressioncited by 42
  11. 2020Kernel regression in high dimensions: Refined analysis beyond double descentcited by 39
  12. 2020Generalization Error of Generalized Linear Models in High Dimensionscited by 25
  13. 2022Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regressioncited by 22
  14. 2019A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learningcited by 20
  15. 2022Mitigating multiple descents: A model-agnostic framework for risk monotonizationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 1; 2025: 4; 2026: 1.

10 papers.

  1. 2023Per-Example Gradient Regularization Improves Learning Signals from Noisy Datacited by 7
  2. 2023DSD2: Can We Dodge Sparse Double Descent and Compress the Neural Network Worry-Free?cited by 6
  3. 2023Can we avoid Double Descent in Deep Neural Networks?cited by 2
  4. 2025Understanding Generalization in Transformers: Error Bounds and Training Dynamics Under Benign and Harmful Overfittingcited by 2
  5. 2024No Free Lunch From Random Feature Ensembles: Scaling Laws and Near-Optimality Conditionscited by 1
  6. 2025High-dimensional ridge regression with random features for non-identically distributed data with a variance profilecited by 1
  7. 2025Multiple Descents in Deep Learning as a Sequence of Order-Chaos Transitions in LSTM Networkscited by 1
  8. 2023The Quest of Finding the Antidote to Sparse Double Descentcited by 0
  9. 2025Gradient Descent Robustly Learns the Intrinsic Dimension of Data in Training Convolutional Neural Networkscited by 0
  10. 2026RAVEN: Frozen Random Graph Reservoirs with Physics-Informed Interaction Fingerprints for Protein-Ligand Binding Affinity Predictioncited by 0