mlsanityData to 14 Oct 2026

HORST: Composing Optimizer Geometries for Sparse Transformer Training

arXiv 2605.21104 ↗First posted 20 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Sparsifying transformers remains a fundamental challenge, as standard optimizers fail to simultaneously encourage sparsity and maintain training stability. Effective adaptive optimizers exhibit an implicit L∞ bias favoring stability, yet, sparsity requires an L1 bias. To integrate sparsity, we propose a composition of optimizer steps, which we cast as non-commutative operators to analyze and combine their optimization geometry in a principled way. This yields HORST (Hyperbolic Operator for Robust Sparse Training), a modular optimizer that inherits stability from adaptive methods while inducing L1 sparsity bias through a hyperbolic mirror map. Our experiments demonstrate its utility for sparse training of transformers on both vision and language tasks. HORST consistently and significantly outperforms AdamW baselines across all sparsity levels, with large gains at higher sparsity.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 63 of its 80 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

63 of 80 references.

References · 63

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  10. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  11. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  12. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
38 more
  1. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  2. 2017Implicit Regularization in Matrix Factorizationcited by 322
  3. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  4. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  5. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  6. 2021Accelerating Sparse Deep Neural Networkscited by 223
  7. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  8. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 183
  9. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 165
  10. 2020Sparse GPU Kernels for Deep Learningcited by 141
  11. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
  12. 2019Implicit Regularization for Optimal Sparse Recoverycited by 80
  13. 2024Modular Duality in Deep Learningcited by 77
  14. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 70
  15. 2024Scalable Optimization in the Modular Normcited by 68
  16. 2025Benchmarking Optimizers for Large Language Model Pretrainingcited by 68
  17. 2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 59
  18. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 49
  19. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 42
  20. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 41
  21. 2025In Search of Adam's Secret Saucecited by 40
  22. 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysiscited by 39
  23. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 39
  24. 2023Dynamic Sparse Training with Structured Sparsitycited by 38
  25. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 36
  26. 2024The Implicit Bias of Adam on Separable Datacited by 36
  27. 2021Implicit Sparse Regularization: The Impact of Depth and Early Stoppingcited by 30
  28. 2022Mirror Descent Maximizes Generalized Margin and Can Be Implemented Efficientlycited by 26
  29. 2022Why Random Pruning Is All We Need to Start Sparsecited by 26
  30. 2019High-Dimensional Linear Regression via Implicit Regularizationcited by 25
  31. 2022Most Activation Functions Can Win the Lottery Without Excessive Depthcited by 21
  32. 2020Learning compositional functions via multiplicative weight updatescited by 20
  33. 2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 18
  34. 2022spred: Solving L1 Penalty with SGDcited by 17
  35. 2022Convolutional and Residual Networks Provably Contain Lottery Ticketscited by 16
  36. 2024Masks, Signs, And Learning Rate Rewindingcited by 15
  37. 2024Mask in the Mirror: Implicit Sparsificationcited by 14
  38. 2025Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetriescited by 14

50 most cited of 63 All 63 (JSON)

Built on by

Not cited yet.