# Inertial Quadratic Majorization Minimization with Application to Kernel Regularized Learning

arXiv: https://arxiv.org/abs/2507.04247 · HTML: https://mlsanity.com/paper/2507.04247/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 74 references that are in the map.
- First posted on arXiv 6 Jul 2025.

## Builds on (most cited first, 17 of 17)

- [Generalization Properties of Learning with Random Features](https://mlsanity.com/paper/1602.04474/) (2016)
- [Adaptive Gradient Descent without Descent](https://mlsanity.com/paper/1910.09529/) (2019)
- [Calculus of the exponent of Kurdyka-Lojasiewicz inequality and its applications to linear convergence of first-order methods](https://mlsanity.com/paper/1602.02915/) (2016)
- [Randomized sketches for kernels: Fast and optimal non-parametric regression](https://mlsanity.com/paper/1501.06195/) (2015)
- [Smoothed Quantile Regression with Large-Scale Inference](https://mlsanity.com/paper/2012.05187/) (2020)
- [Super-Universal Regularized Newton Method](https://mlsanity.com/paper/2208.05888/) (2022)
- [Smoothing quantile regressions](https://mlsanity.com/paper/1905.08535/) (2019)
- [An Inertial Block Majorization Minimization Framework for Nonsmooth Nonconvex Optimization](https://mlsanity.com/paper/2010.12133/) (2020)
- [A proximal difference-of-convex algorithm with extrapolation](https://mlsanity.com/paper/1612.06265/) (2016)
- [An inertial forward-backward algorithm for monotone inclusions](https://mlsanity.com/paper/1403.3522/) (2014)
- [Inertial Krasnoselskii-Mann Iterations](https://mlsanity.com/paper/2210.03791/) (2022)
- [Block Alternating Bregman Majorization Minimization with Extrapolation](https://mlsanity.com/paper/2107.04395/) (2021)
- [Optimal Kernel Quantile Learning with Random Features](https://mlsanity.com/paper/2408.13591/) (2024)
- [Linear Convergence of Proximal Gradient Algorithm with Extrapolation for a Class of Nonconvex Nonsmooth Minimization Problems](https://mlsanity.com/paper/1512.09302/) (2015)
- [A Proximal Distance Algorithm for Likelihood-Based Sparse Covariance Estimation](https://mlsanity.com/paper/2109.04497/) (2021)
- [A Unified Algorithm for Penalized Convolution Smoothed Quantile Regression](https://mlsanity.com/paper/2205.02432/) (2022)
- [Tactics for Improving Least Squares Estimation](https://mlsanity.com/paper/2501.02475/) (2025)

## Abstract (arXiv)

First-order methods in convex optimization offer low per-iteration cost but often suffer from slow convergence, while second-order methods achieve fast local convergence at the expense of costly Hessian inversions. In this paper, we highlight a middle ground: minimizing a quadratic majorant with fixed curvature at each iteration. This strategy strikes a balance between per-iteration cost and convergence speed, and crucially allows the reuse of matrix decompositions, such as Cholesky or spectral decompositions, across iterations and varying regularization parameters. We introduce the Quadratic Majorization Minimization with Extrapolation (QMME) framework and establish its sequential convergence properties under standard assumptions. The new perspective of our analysis is to center the arguments around the induced norm of the curvature matrix H. To demonstrate practical advantages, we apply QMME to large-scale kernel regularized learning problems. In particular, we propose a novel Sylvester equation modelling technique for kernel multinomial regression. In Julia-based experiments, QMME compares favorably against various established first- and second-order methods. Furthermore, we demonstrate that our algorithms complement existing kernel approximation techniques through more efficiently handling sketching matrices with large projection dimensions. Our numerical experiments and real data analysis are available and fully reproducible at https://github.com/qhengncsu/QMME.jl.