Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks

We consider one-hidden layer neural networks trained in the feature-learning regime using gradient descent, and relate the output of the finite-width network f_hatρtm to its infinite-width counterpart f_ρtMF, which evolves in the mean-field dynamics. While constant-time horizon bounds for |f_ρtMF - f_hatρtm| may be obtained via standard Gr"onwall estimates, the long-time behavior of the fluctuation is a more delicate matter. Uniform-in-time bounds often rely on (local) strong convexity in the landscape or Logarithmic Sobolev inequalities present in noisy gradient dynamics. In this work, we establish non-asymptotic weak propagation-of-chaos that holds uniformly in time, obtained by exploiting instead the convergence rate of the mean-field deterministic Wasserstein-gradient-flow dynamics. Specifically, denoting by Lt the mean-field excess MSE loss at time t and m the number of neurons, under standard regularity assumptions and the condition int0^∞ Lt1/2 dt =O(log d), we obtain the uniform in time bound |f_ρtMF- f_hatρtm|2 lesssim poly(d) m-min(1,c/6) whenever Lt lesssim t-c. Our result holds in a noiseless setting and does not make any assumptions on the geometry of the landscape near the optimum, and extends seamlessly to other forms of discretization, including finite number of samples and time discretization. A key takeaway of our result is that whenever the convergence rate of the mean-field, population-loss dynamics is faster than t-2, we can attain a loss of ε with only poly(d/ε) neurons, training samples, and GD steps.