# End-to-End Large Portfolio Optimization for Variance Minimization with Neural Networks through Covariance Cleaning

arXiv: https://arxiv.org/abs/2507.01918 · HTML: https://mlsanity.com/paper/2507.01918/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 84 references that are in the map.
- First posted on arXiv 2 Jul 2025.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [Understanding the Difficulty of Training Transformers](https://mlsanity.com/paper/2004.08249/) (2020)
- [Universal Approximation of Functions on Sets](https://mlsanity.com/paper/2107.01959/) (2021)
- [Distributionally Robust End-to-End Portfolio Construction](https://mlsanity.com/paper/2206.05134/) (2022)
- [Non-linear shrinkage of the price return covariance matrix is far from optimal for portfolio optimisation](https://mlsanity.com/paper/2112.07521/) (2021)
- [Deep Deterministic Portfolio Optimization](https://mlsanity.com/paper/2003.06497/) (2020)
- [Estimation of Large Financial Covariances: A Cross-Validation Approach](https://mlsanity.com/paper/2012.05757/) (2020)
- [A Deep Learning Framework for Medium-Term Covariance Forecasting in Multi-Asset Portfolios](https://mlsanity.com/paper/2503.01581/) (2025)
- [Covariance matrix filtering with bootstrapped hierarchies](https://mlsanity.com/paper/2003.05807/) (2020)
- [Reactive Global Minimum Variance Portfolios with k-BAHC covariance cleaning](https://mlsanity.com/paper/2005.08703/) (2020)
- [Two-step estimators of high dimensional correlation matrices](https://mlsanity.com/paper/2212.14650/) (2022)

## Built on by (most cited first, 5 of 5)

- [Agentic Trading: When LLM Agents Meet Financial Markets](https://mlsanity.com/paper/2605.19337/) (2026)
- [Neural Network-Driven Volatility Drag Mitigation under Aggressive Leverage](https://mlsanity.com/paper/2607.23068/) (2026)
- [Integrated Prediction and Multi-period Portfolio Optimization](https://mlsanity.com/paper/2512.11273/) (2025)
- [Physics-Informed Singular-Value Learning for Cross-Covariances Forecasting in Financial Markets](https://mlsanity.com/paper/2601.07687/) (2026)
- [End-to-End Neural Shrinkage of Indefinite Pairwise Correlation Matrices for Small-Cap-Inclusive Portfolios](https://mlsanity.com/paper/2608.30446/) (2026)

## Abstract (arXiv)

We develop a rotation-invariant neural network that provides the global minimum-variance portfolio by jointly learning how to lag-transform historical returns and marginal volatilities and how to regularise the eigenvalues of large equity covariance matrices. This explicit mathematical mapping offers clear interpretability of each module's role, so the model cannot be regarded as a pure black box. The architecture mirrors the analytical form of the global minimum-variance solution yet remains agnostic to dimension, so a single model can be calibrated on panels of a few hundred stocks and applied, without retraining, to one thousand US equities, a cross-sectional jump that indicates robust generalization capability. The loss function is the future short-term realized minimum variance and is optimized end-to-end on real returns. In out-of-sample tests from January 2000 to December 2024, the estimator delivers systematically lower realized volatility, smaller maximum drawdowns, and higher Sharpe ratios than the best competitors, including state-of-the-art non-linear shrinkage, and these advantages persist across both short and long evaluation horizons despite the model's training focus is short-term. Furthermore, although the model is trained end-to-end to produce an unconstrained minimum-variance portfolio, we show that its learned covariance representation can be used in general optimizers under long-only constraints with virtually no loss in its performance advantage over competing estimators. These advantages persist when the strategy is executed under a highly realistic implementation framework that models market orders at the auctions, empirical slippage, exchange fees, and financing charges for leverage, and they remain stable during episodes of acute market stress.