# Multiple Descent in the Multiple Random Feature Model

arXiv: https://arxiv.org/abs/2208.09897 · HTML: https://mlsanity.com/paper/2208.09897/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 21 Aug 2022.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [Two models of double descent for weak features](https://mlsanity.com/paper/1903.07571/) (2019)
- [Optimal Regularization Can Mitigate Double Descent](https://mlsanity.com/paper/2003.01897/) (2020)
- [The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization](https://mlsanity.com/paper/2008.06786/) (2020)
- [On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regression](https://mlsanity.com/paper/2006.05800/) (2020)
- [Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime](https://mlsanity.com/paper/2003.01054/) (2020)
- [The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Training](https://mlsanity.com/paper/2007.12826/) (2020)
- [Scaling description of generalization with number of parameters in deep learning](https://mlsanity.com/paper/1901.01608/) (2019)
- [A Model of Double Descent for High-dimensional Binary Linear Classification](https://mlsanity.com/paper/1911.05822/) (2019)
- [Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regime](https://mlsanity.com/paper/2004.12019/) (2020)
- [Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains](https://mlsanity.com/paper/2007.11207/) (2020)
- [Understanding Double Descent Requires a Fine-Grained Bias-Variance Decomposition](https://mlsanity.com/paper/2011.03321/) (2020)
- [Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Data](https://mlsanity.com/paper/2202.05928/) (2022)
- [A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descent](https://mlsanity.com/paper/2006.05013/) (2020)
- [Triple descent and the two kinds of overfitting: Where & why do they appear?](https://mlsanity.com/paper/2006.03509/) (2020)
- [Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixtures](https://mlsanity.com/paper/2104.13628/) (2021)
- [Multiple Descent: Design Your Own Generalization Curve](https://mlsanity.com/paper/2008.01036/) (2020)
- [Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regression](https://mlsanity.com/paper/2204.10425/) (2022)
- [Kernel regression in high dimensions: Refined analysis beyond double descent](https://mlsanity.com/paper/2010.02681/) (2020)
- [Generalization Error of Generalized Linear Models in High Dimensions](https://mlsanity.com/paper/2005.00180/) (2020)
- [Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regression](https://mlsanity.com/paper/2205.14846/) (2022)
- [A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learning](https://mlsanity.com/paper/1912.00827/) (2019)
- [Mitigating multiple descents: A model-agnostic framework for risk monotonization](https://mlsanity.com/paper/2205.12937/) (2022)

## Built on by (most cited first, 10 of 10)

- [Per-Example Gradient Regularization Improves Learning Signals from Noisy Data](https://mlsanity.com/paper/2303.17940/) (2023)
- [DSD2: Can We Dodge Sparse Double Descent and Compress the Neural Network Worry-Free?](https://mlsanity.com/paper/2303.01213/) (2023)
- [Can we avoid Double Descent in Deep Neural Networks?](https://mlsanity.com/paper/2302.13259/) (2023)
- [Understanding Generalization in Transformers: Error Bounds and Training Dynamics Under Benign and Harmful Overfitting](https://mlsanity.com/paper/2502.12508/) (2025)
- [High-dimensional ridge regression with random features for non-identically distributed data with a variance profile](https://mlsanity.com/paper/2504.03035/) (2025)
- [No Free Lunch From Random Feature Ensembles: Scaling Laws and Near-Optimality Conditions](https://mlsanity.com/paper/2412.05418/) (2024)
- [Multiple Descents in Deep Learning as a Sequence of Order-Chaos Transitions in LSTM Networks](https://mlsanity.com/paper/2505.20030/) (2025)
- [The Quest of Finding the Antidote to Sparse Double Descent](https://mlsanity.com/paper/2308.16596/) (2023)
- [Gradient Descent Robustly Learns the Intrinsic Dimension of Data in Training Convolutional Neural Networks](https://mlsanity.com/paper/2504.08628/) (2025)
- [RAVEN: Frozen Random Graph Reservoirs with Physics-Informed Interaction Fingerprints for Protein-Ligand Binding Affinity Prediction](https://mlsanity.com/paper/2608.09099/) (2026)

## Abstract (arXiv)

Recent works have demonstrated a double descent phenomenon in over-parameterized learning. Although this phenomenon has been investigated by recent works, it has not been fully understood in theory. In this paper, we investigate the multiple descent phenomenon in a class of multi-component prediction models. We first consider a ''double random feature model'' (DRFM) concatenating two types of random features, and study the excess risk achieved by the DRFM in ridge regression. We calculate the precise limit of the excess risk under the high dimensional framework where the training sample size, the dimension of data, and the dimension of random features tend to infinity proportionally. Based on the calculation, we further theoretically demonstrate that the risk curves of DRFMs can exhibit triple descent. We then provide a thorough experimental study to verify our theory. At last, we extend our study to the ''multiple random feature model'' (MRFM), and show that MRFMs ensembling K types of random features may exhibit (K+1)-fold descent. Our analysis points out that risk curves with a specific number of descent generally exist in learning multi-component prediction models.