Sharp Approximation Rates for Neural Networks with Affine Latent Parameterizations

Many parameter-efficient methods generate the parameters of a large neural network from a low-dimensional latent representation. Given an architecture Phi with P_Phi parameter slots, we write boldsymbolθf=G(boldsymbolxif), where GcolonℝM→ℝP_Phi is a parameter generator and boldsymbolxif∈ℝM is a latent representation of the target function f. The architecture Phi and the generator G are shared across the entire target class, while each target f is represented by its own latent vector boldsymbolxif, with Phi_G(boldsymbolxif) approximating f. This framework encompasses hypernetworks, low-dimensional parameterizations, parameter-efficient adaptation, and model compression. Understanding the tradeoff between the latent dimension M and the network budget P is therefore fundamental to characterizing the expressive efficiency of these methods. We study this tradeoff for affine generators and fully connected ReLU architectures. More precisely, optimizing jointly over architectures Phi satisfying P_Phi≤ P and affine generators G:ℝM→ ℝP_Phi, we prove that the optimal worst-case uniform approximation error over the unit ball of α-H"older functions on [0,1]d, where 0<α≤1, has the sharp order bigl(PminM,Pbigr)-α/d. In particular, our result shows that even a fixed-dimensional latent space suffices to achieve vanishing approximation error as the network budget increases.