Moreau envelope and proximal-point methods under the lens of high-order regularization

This paper is devoted to investigating the fundamental properties of the high-order proximal operator (HOPE) and the high-order Moreau envelope (HOME) in the nonconvex setting, where the quadratic regularization (p=2) is replaced by a p-order regularizer with p > 1. After establishing several basic properties of HOPE and HOME, we study the differentiability and weak smoothness of HOME under q-prox-regularity with q ≥ 2 and p-calmness for p ∈ (1,2] and 2 ≤ p ≤ q. Furthermore, we propose a high-order proximal-point algorithm (HiPPA) and analyze the convergence of the generated sequence to proximal fixed points. Our results pave the way for the development of a high-order smoothing theory with p>1 that can lead to new algorithmic advances in the nonconvex setting. To illustrate this potential for nonsmooth and nonconvex optimization, we apply HiPPA to the Nesterov-Chebyshev-Rosenbrock functions.