Theoretical aspects in penalty hyperparameters optimization

Learning processes are useful methodologies able to improve knowledge of real phenomena. These are often dependent on hyperparameters, variables set before the training process and regulating the learning procedure. Hyperparameters optimization problem is an open issue in learning approaches since it can strongly affect any real data analysis. They are usually selected using Grid-Search or Cross Validation techniques. No automatic tuning procedure exists especially if we focus on an unsupervised learning scenario. This study aims to assess some theoretical considerations for tuning penalty hyperparameters in optimization problems. It considers a bi-level formulation tuning problem in an unsupervised context, by using Gradient-based methods. Suitable conditions for the existence of a minimizer in an infinite-dimensional Hilbert space are outlined, together with some theoretical results, applicable in all those situations when it is unnecessary or not possible obtaining an exact minimizer. An iterative algorithmic strategy is considered, equipped with a stopping criterion via Ekeland's variational principle.