Covariance-Aware Transformers for Quadratic Programming and Decision Making
Not on the map. Why?
Abstract (arXiv)
We explore the use of transformers for solving quadratic programs and how this capability benefits decision-making problems that involve covariance matrices. We first show that the linear attention mechanism can provably solve unconstrained QPs by tokenizing the matrix variables (e.g.~A of the objective frac12x^top Ax+b^top x) row-by-row and emulating gradient descent iterations. Furthermore, by incorporating MLPs, a transformer block can solve (i) ℓ1-penalized QPs by emulating iterative soft-thresholding and (ii) ℓ1-constrained QPs when equipped with an additional feedback loop. Our theory motivates us to introduce Time2Decide: a generic method that enhances a time series foundation model (TSFM) by explicitly feeding the covariance matrix between the variates. We empirically find that Time2Decide uniformly outperforms the base TSFM model for the classical portfolio optimization problem that admits an ℓ1-constrained QP formulation. Remarkably, Time2Decide also outperforms the classical "Predict-then-Optimize (PtO)" procedure, where we first forecast the returns and then explicitly solve a constrained QP, in suitable settings. Our results demonstrate that transformers benefit from explicit use of second-order statistics, and this can enable them to effectively solve complex decision-making problems, like portfolio construction, in one forward pass.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 30 references that are in the map. First posted on arXiv 16 Feb 2026.
Builds on
25 of 30 references.
References · 25
- 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
- 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
- 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,654
- 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,200
- 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
- 2019Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecastingcited by 748
- 2022Transformers learn in-context by gradient descentcited by 714
- 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
- 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 588
- 2017Smart "Predict, then Optimize"cited by 332
- 2023Trained Transformers Learn Linear Models In-Contextcited by 331
- 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
13 more
- 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
- 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
- 2020Solving Mixed Integer Programs Using Neural Networkscited by 178
- 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
- 2021Long-Range Transformers for Dynamic Spatiotemporal Forecastingcited by 46
- 2022On Representing Linear Programs by Graph Neural Networkscited by 39
- 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 36
- 2025TimePFN: Effective Multivariate Time Series Forecasting with Synthetic Datacited by 27
- 2024Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyondcited by 22
- 2024On the Training Convergence of Transformers for In-Context Classification of Gaussian Mixturescited by 15
- 2025Test-Time Training Provably Improves Transformers as In-context Learnerscited by 15
- 2024Differentiation Through Black-Box Quadratic Programming Solverscited by 10
- 2022Neural Optimization Machine: A Neural Network Approach for Optimizationcited by 4
Built on by
Not cited yet.