Frank-Wolfe Network: An Interpretable Deep Structure for Non-Sparse Coding

The problem of Lp-norm constrained coding is to convert signal into code that lies inside an Lp-ball and most faithfully reconstructs the signal. Previous works under the name of sparse coding considered the cases of L0 and L1 norms. The cases with p>1 values, i.e. non-sparse coding studied in this paper, remain a difficulty. We propose an interpretable deep structure namely Frank-Wolfe Network (F-W Net), whose architecture is inspired by unrolling and truncating the Frank-Wolfe algorithm for solving an Lp-norm constrained problem with p≥ 1. We show that the Frank-Wolfe solver for the Lp-norm constraint leads to a novel closed-form nonlinear unit, which is parameterized by p and termed poolp. The poolp unit links the conventional pooling, activation, and normalization operations, making F-W Net distinct from existing deep networks either heuristically designed or converted from projected gradient descent algorithms. We further show that the hyper-parameter p can be made learnable instead of pre-chosen in F-W Net, which gracefully solves the non-sparse coding problem even with unknown p. We evaluate the performance of F-W Net on an extensive range of simulations as well as the task of handwritten digit recognition, where F-W Net exhibits strong learning capability. We then propose a convolutional version of F-W Net, and apply the convolutional F-W Net into image denoising and super-resolution tasks, where F-W Net all demonstrates impressive effectiveness, flexibility, and robustness.