# Robust Regression via Model Based Methods

arXiv: https://arxiv.org/abs/2106.10759 · HTML: https://mlsanity.com/paper/2106.10759/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 52 references that are in the map.
- First posted on arXiv 20 Jun 2021.

## Builds on (most cited first, 12 of 12)

- [Stochastic model-based minimization of weakly convex functions](https://mlsanity.com/paper/1803.06523/) (2018)
- [Error bounds, quadratic growth, and linear convergence of proximal methods](https://mlsanity.com/paper/1602.06661/) (2016)
- [Efficiency of minimizing compositions of convex functions and smooth maps](https://mlsanity.com/paper/1605.00125/) (2016)
- [Stochastic Methods for Composite and Weakly Convex Optimization Problems](https://mlsanity.com/paper/1703.08570/) (2017)
- [Convergence of a Stochastic Gradient Method with Momentum for Non-Smooth Non-Convex Optimization](https://mlsanity.com/paper/2002.05466/) (2020)
- [Proximally Guided Stochastic Subgradient Method for Nonsmooth, Nonconvex Problems](https://mlsanity.com/paper/1707.03505/) (2017)
- [Online Robust Regression via SGD on the l1 loss](https://mlsanity.com/paper/2007.00399/) (2020)
- [Multi-Target Prediction: A Unifying View on Problems and Methods](https://mlsanity.com/paper/1809.02352/) (2018)
- [Stochastic Variance-Reduced ADMM](https://mlsanity.com/paper/1604.07070/) (2016)
- [Accelerated Variance Reduced Stochastic ADMM](https://mlsanity.com/paper/1707.03190/) (2017)
- [Non-smooth Non-convex Bregman Minimization: Unification and new Algorithms](https://mlsanity.com/paper/1707.02278/) (2017)
- [Model Function Based Conditional Gradient Method with Armijo-like Line Search](https://mlsanity.com/paper/1901.08087/) (2019)

## Abstract (arXiv)

The mean squared error loss is widely used in many applications, including auto-encoders, multi-target regression, and matrix factorization, to name a few. Despite computational advantages due to its differentiability, it is not robust to outliers. In contrast, l_p norms are known to be robust, but cannot be optimized via, e.g., stochastic gradient descent, as they are non-differentiable. We propose an algorithm inspired by so-called model-based optimization (MBO) [35, 36], which replaces a non-convex objective with a convex model function and alternates between optimizing the model function and updating the solution. We apply this to robust regression, proposing SADM, a stochastic variant of the Online Alternating Direction Method of Multipliers (OADM) [50] to solve the inner optimization in MBO. We show that SADM converges with the rate O(log T/T). Finally, we demonstrate experimentally (a) the robustness of l_p norms to outliers and (b) the efficiency of our proposed model-based algorithms in comparison with gradient methods on autoencoders and multi-target regression.