Error bound and exact penalty method for optimization problems with nonnegative orthogonal constraint

This paper is concerned with a class of optimization problems with the nonnegative orthogonal constraint, in which the objective function is L-smooth on an open set containing the Stiefel manifold rm St(n,r). We derive a locally Lipschitzian error bound for the feasible points without zero rows when n>r>1, and when n>r=1 or n=r achieve a global Lipschitzian error bound. Then, we show that the penalty problem induced by the elementwise ℓ1-norm distance to the nonnegative cone is a global exact penalty, and so is the one induced by its Moreau envelope under a lower second-order calmness of the objective function. A practical penalty algorithm is developed by solving approximately a series of smooth penalty problems with a retraction-based nonmonotone line-search proximal gradient method, and any cluster point of the generated sequence is shown to be a stationary point of the original problem. Numerical comparisons with the ALM citepWen13 and the exact penalty method citepJiangM22 indicate that our penalty method has an advantage in terms of the quality of solutions despite taking a little more time.