# Orthant Based Proximal Stochastic Gradient Method for ℓ1-Regularized Optimization

arXiv: https://arxiv.org/abs/2004.03639 · HTML: https://mlsanity.com/paper/2004.03639/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Apr 2020.

## Built on by (most cited first, 20 of 20)

- [Only Train Once: A One-Shot Neural Network Training And Pruning Framework](https://mlsanity.com/paper/2107.07467/) (2021)
- [CDFI: Compression-Driven Network Design for Frame Interpolation](https://mlsanity.com/paper/2103.10559/) (2021)
- [LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery](https://mlsanity.com/paper/2310.18356/) (2023)
- [OTOV2: Automatic, Generic, User-Friendly](https://mlsanity.com/paper/2303.06862/) (2023)
- [The Efficiency Spectrum of Large Language Models: An Algorithmic Survey](https://mlsanity.com/paper/2312.00678/) (2023)
- [Automatic Joint Structured Pruning and Quantization for Efficient Neural Network Training and Compression](https://mlsanity.com/paper/2502.16638/) (2025)
- [ST-MFNet Mini: Knowledge Distillation-Driven Frame Interpolation](https://mlsanity.com/paper/2302.08455/) (2023)
- [Compressing Deep Image Super-resolution Models](https://mlsanity.com/paper/2401.00523/) (2023)
- [Sparsity-guided Network Design for Frame Interpolation](https://mlsanity.com/paper/2209.04551/) (2022)
- [OTOv3: Automatic Architecture-Agnostic Neural Network Training and Compression from Structured Pruning to Erasing Operators](https://mlsanity.com/paper/2312.09411/) (2023)
- [HESSO: Towards Automatic Efficient and User Friendly Any Neural Network Training and Pruning](https://mlsanity.com/paper/2409.09085/) (2024)
- [A multiobjective continuation method to compute the regularization path of deep neural networks](https://mlsanity.com/paper/2308.12044/) (2023)
- [ONNXPruner: ONNX-Based General Model Pruning Adapter](https://mlsanity.com/paper/2404.08016/) (2024)
- [WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inference](https://mlsanity.com/paper/2505.19427/) (2025)
- [PathProx: A Proximal Gradient Algorithm for Weight Decay Regularized Deep Neural Networks](https://mlsanity.com/paper/2210.03069/) (2022)
- [FSCNN: A Fast Sparse Convolution Neural Network Inference System](https://mlsanity.com/paper/2212.08815/) (2022)
- [SMOF: Squeezing More Out of Filters Yields Hardware-Friendly CNN Pruning](https://mlsanity.com/paper/2110.10842/) (2021)
- [RankDVQA-mini: Knowledge Distillation-Driven Deep Video Quality Assessment](https://mlsanity.com/paper/2312.08864/) (2023)
- [DaBiT: Depth and Blur informed Transformer for Video Focal Deblurring](https://mlsanity.com/paper/2407.01230/) (2024)
- [PocketDVDNet: Realtime Video Denoising for Real Camera Noise](https://mlsanity.com/paper/2601.16780/) (2026)

## Abstract (arXiv)

Sparsity-inducing regularization problems are ubiquitous in machine learning applications, ranging from feature selection to model compression. In this paper, we present a novel stochastic method -- Orthant Based Proximal Stochastic Gradient Method (OBProx-SG) -- to solve perhaps the most popular instance, i.e., the l1-regularized problem. The OBProx-SG method contains two steps: (i) a proximal stochastic gradient step to predict a support cover of the solution; and (ii) an orthant step to aggressively enhance the sparsity level via orthant face projection. Compared to the state-of-the-art methods, e.g., Prox-SG, RDA and Prox-SVRG, the OBProx-SG not only converges to the global optimal solutions (in convex scenario) or the stationary points (in non-convex scenario), but also promotes the sparsity of the solutions substantially. Particularly, on a large number of convex problems, OBProx-SG outperforms the existing methods comprehensively in the aspect of sparsity exploration and objective values. Moreover, the experiments on non-convex deep neural networks, e.g., MobileNetV1 and ResNet18, further demonstrate its superiority by achieving the solutions of much higher sparsity without sacrificing generalization accuracy.