# Mallows-type model averaging: Non-asymptotic analysis and all-subset combination

arXiv: https://arxiv.org/abs/2505.02637 · HTML: https://mlsanity.com/paper/2505.02637/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 109 references that are in the map.
- First posted on arXiv 5 May 2025.

## Builds on (most cited first, 17 of 17)

- [Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks](https://mlsanity.com/paper/1901.08584/) (2019)
- [The Implicit Regularization of Ordinary Least Squares Ensembles](https://mlsanity.com/paper/1910.04743/) (2019)
- [Subsample Ridge Ensembles: Equivalences and Generalized Cross-Validation](https://mlsanity.com/paper/2304.13016/) (2023)
- [Asymptotically free sketched ridge ensembles: Risks, cross-validation, and tuning](https://mlsanity.com/paper/2310.04357/) (2023)
- [Optimal Weighted Random Forests](https://mlsanity.com/paper/2305.10042/) (2023)
- [Corrected generalized cross-validation for finite ensembles of penalized estimators](https://mlsanity.com/paper/2310.01374/) (2023)
- [Aggregation of Affine Estimators](https://mlsanity.com/paper/1311.2799/) (2013)
- [Extrapolated cross-validation for randomized ensembles](https://mlsanity.com/paper/2302.13511/) (2023)
- [The cost-free nature of optimally tuning Tikhonov regularizers and other ordered smoothers](https://mlsanity.com/paper/1905.12517/) (2019)
- [Ensemble linear interpolators: The role of ensembling](https://mlsanity.com/paper/2309.03354/) (2023)
- [On the design-dependent suboptimality of the Lasso](https://mlsanity.com/paper/2402.00382/) (2024)
- [Unified optimal model averaging with a general loss function based on cross-validation](https://mlsanity.com/paper/2412.09804/) (2024)
- [Sparsity Oriented Importance Learning for High-dimensional Linear Regression](https://mlsanity.com/paper/1608.00629/) (2016)
- [Optimal integrating learning for split questionnaire design type data](https://mlsanity.com/paper/2108.02905/) (2021)
- [Complete Subset Averaging with Many Instruments](https://mlsanity.com/paper/1811.08083/) (2018)
- [Error Reduction from Stacked Regressions](https://mlsanity.com/paper/2309.09880/) (2023)
- [Stability and L2-penalty in Model Averaging](https://mlsanity.com/paper/2311.13827/) (2023)

## Abstract (arXiv)

Model averaging (MA) and ensembling play a crucial role in statistical and machine learning practice. When multiple candidate models are considered, MA techniques can be used to weight and combine them, often resulting in improved predictive accuracy and better estimation stability compared to model selection (MS) methods. In this paper, we address two important problems in combining least squares estimators from both theoretical and practical perspectives. We first establish several oracle inequalities for least squares MA via minimizing a Mallows' Cp criterion under an arbitrary candidate model set. Compared to existing studies, these oracle inequalities yield faster excess risk and directly imply the asymptotic optimality of the resulting MA estimators under milder conditions. Moreover, we consider candidate model construction and investigate the problem of optimal all-subset combination for least squares estimators, which remains an important yet not fully understood topic in the existing literature. We show that there exists a fundamental limit to achieving the optimal all-subset MA risk. To attain this limit, we propose a novel Mallows-type MA procedure based on a dimension-adaptive Cp criterion. The implicit ensembling effects of several MS procedures are also revealed and discussed. We conduct several numerical experiments to support our theoretical findings and demonstrate the effectiveness of the proposed Mallows-type MA estimator.