# Efficient Meta Subspace Optimization

arXiv: https://arxiv.org/abs/2110.14920 · HTML: https://mlsanity.com/paper/2110.14920/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 28 Oct 2021.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Learned Optimizers that Scale and Generalize](https://mlsanity.com/paper/1703.04813/) (2017)
- [On Empirical Comparisons of Optimizers for Deep Learning](https://mlsanity.com/paper/1910.05446/) (2019)
- [Understanding and correcting pathologies in the training of learned optimizers](https://mlsanity.com/paper/1810.10180/) (2018)
- [Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves](https://mlsanity.com/paper/2009.11243/) (2020)
- [Learning Gradient Descent: Better Generalization and Longer Horizons](https://mlsanity.com/paper/1703.03633/) (2017)
- [Learning to Optimize Neural Nets](https://mlsanity.com/paper/1703.00441/) (2017)
- [Learning an Adaptive Learning Rate Schedule](https://mlsanity.com/paper/1909.09712/) (2019)
- [Reinforcement Learning for Learning Rate Control](https://mlsanity.com/paper/1705.11159/) (2017)
- [Primal-Dual Sequential Subspace Optimization for Saddle-point Problems](https://mlsanity.com/paper/2008.09149/) (2020)
- [SEBOOST - Boosting Stochastic Learning Using Subspace Optimization Techniques](https://mlsanity.com/paper/1609.00629/) (2016)

## Built on by (most cited first, 1 of 1)

- [Adaptive Consensus Gradients Aggregation for Scaled Distributed Training](https://mlsanity.com/paper/2411.03742/) (2024)

## Abstract (arXiv)

Subspace optimization methods have the attractive property of reducing large-scale optimization problems to a sequence of low-dimensional subspace optimization problems. However, existing subspace optimization frameworks adopt a fixed update policy of the subspace and therefore appear to be sub-optimal. In this paper, we propose a new Meta Subspace Optimization (MSO) framework for large-scale optimization problems, which allows to determine the subspace matrix at each optimization iteration. In order to remain invariant to the optimization problem's dimension, we design an efficient meta optimizer based on very low-dimensional subspace optimization coefficients, inducing a rule-based method that can significantly improve performance. Finally, we design and analyze a reinforcement learning (RL) procedure based on the subspace optimization dynamics whose learnt policies outperform existing subspace optimization methods.