# Manifold constrained steepest descent for smooth and closed-set optimization

arXiv: https://arxiv.org/abs/2601.21487 · HTML: https://mlsanity.com/paper/2601.21487/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 67 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Builds on (most cited first, 24 of 24)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming](https://mlsanity.com/paper/1309.5549/) (2013)
- [Sorting out Lipschitz function approximation](https://mlsanity.com/paper/1811.05381/) (2018)
- [Riemannian Adaptive Optimization Methods](https://mlsanity.com/paper/1810.00760/) (2018)
- [Old Optimizer, New Norm: An Anthology](https://mlsanity.com/paper/2409.20325/) (2024)
- [Training Deep Learning Models with Norm-Constrained LMOs](https://mlsanity.com/paper/2502.07529/) (2025)
- [The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm](https://mlsanity.com/paper/2505.16932/) (2025)
- [Best Subset Selection via a Modern Optimization Lens](https://mlsanity.com/paper/1507.03133/) (2015)
- [Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks](https://mlsanity.com/paper/1709.06079/) (2017)
- [Controlled LLM Training on Spectral Sphere](https://mlsanity.com/paper/2601.08393/) (2026)
- [An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variants](https://mlsanity.com/paper/2510.09827/) (2025)
- [Optimal Rates of Convergence for Noisy Sparse Phase Retrieval via Thresholded Wirtinger Flow](https://mlsanity.com/paper/1506.03382/) (2015)
- [Riemannian preconditioning](https://mlsanity.com/paper/1405.6055/) (2014)
- [Solving Large-Scale Sparse PCA to Certifiable (Near) Optimality](https://mlsanity.com/paper/2005.05195/) (2020)
- [Convergence results for projected line-search methods on varieties of low-rank matrices via Lojasiewicz inequality](https://mlsanity.com/paper/1402.5284/) (2014)
- [A Preconditioned Riemannian Gradient Descent Algorithm for Low-Rank Matrix Recovery](https://mlsanity.com/paper/2305.02543/) (2023)
- [New Riemannian preconditioned algorithms for tensor completion via polyadic decomposition](https://mlsanity.com/paper/2101.11108/) (2021)
- [A Fast and Provable Algorithm for Sparse Phase Retrieval](https://mlsanity.com/paper/2309.02046/) (2023)
- [Projected gradient descent accumulates at Bouligand stationary points](https://mlsanity.com/paper/2403.02530/) (2024)
- [Low-rank optimization methods based on projected projected-gradient descent that accumulate at Bouligand stationary points](https://mlsanity.com/paper/2201.03962/) (2022)
- [An apocalypse-free first-order low-rank optimization algorithm with at most one rank reduction attempt per iteration](https://mlsanity.com/paper/2208.12051/) (2022)
- [Computing Bouligand stationary points efficiently in low-rank optimization](https://mlsanity.com/paper/2409.12298/) (2024)
- [First-order optimization on stratified sets](https://mlsanity.com/paper/2303.16040/) (2023)

## Built on by (most cited first, 8 of 8)

- [Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers](https://mlsanity.com/paper/2605.18106/) (2026)
- [Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds](https://mlsanity.com/paper/2605.09238/) (2026)
- [SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales](https://mlsanity.com/paper/2607.20548/) (2026)
- [Convergence of Spectral Descent for Non-smooth Optimization](https://mlsanity.com/paper/2605.26977/) (2026)
- [Demystifying Manifold Constraints in LLM Pre-training](https://mlsanity.com/paper/2605.04418/) (2026)
- [Learned Subspace Compression for Communication-Efficient Pipeline Parallelism](https://mlsanity.com/paper/2606.05484/) (2026)
- [Different Layers, Different Manifolds: Module-Wise Weight-Space Geometry in Transformer Optimization](https://mlsanity.com/paper/2606.13276/) (2026)
- [Muon on the Stiefel Manifold Admits an Exact Closed-Form Update](https://mlsanity.com/paper/2608.06218/) (2026)

## Abstract (arXiv)

We study minimization of smooth functions over feasible sets that have smooth embedded-manifold structure throughout or only on selected regions, using linear minimization oracles (LMOs) to determine search directions under user-chosen norms. Restricting an LMO to a tangent space, however, can require an iterative inner solve. We propose Manifold Constrained Steepest Descent (MCSD) and a tangent-projected variant, MCSD-TP, which avoid solving tangent-space LMO subproblems iteratively. The spectral-norm specialization of MCSD on the Stiefel manifold yields SPEL, which admits an efficient implementation using matrix-sign computations. Under standard regularity assumptions, both methods attain an (O(log T/√ T)) best-iterate stationarity bound. For closed feasible sets, the hybrid MCSD--PGD combines either smooth method with projected gradient descent; under the corresponding local smoothness condition, every accumulation point is Bouligand stationary. Experiments on Stiefel-constrained PCA, weighted low-rank approximation, and sparse phase retrieval illustrate the proposed methods.