An Over-parameterized Exponential Regression

Over the past few years, there has been a significant amount of research focused on studying the ReLU activation function, with the aim of achieving neural network convergence through over-parametrization. However, recent developments in the field of Large Language Models (LLMs) have sparked interest in the use of exponential activation functions, specifically in the attention mechanism. Mathematically, we define the neural function F: ℝd × m × ℝd rightarrow ℝ using an exponential activation function. Given a set of data points with labels (x1, y1), (x2, y2), dots, (xn, yn) subset ℝd × ℝ where n denotes the number of the data. Here F(W(t),x) can be expressed as F(W(t),x) := sumr=1m ar exp(langle wr, x rangle), where m represents the number of neurons, and wr(t) are weights at time t. It's standard in literature that ar are the fixed weights and it's never changed during the training. We initialize the weights W(0) ∈ ℝd × m with random Gaussian distributions, such that wr(0) ~ N(0, Id) and initialize ar from random sign distribution for each r ∈ [m]. Using the gradient descent algorithm, we can find a weight W(T) such that | F(W(T), X) - y |2 ≤ ε holds with probability 1-δ, where ε ∈ (0,0.1) and m = Omega(n2+o(1)log(n/δ)). To optimize the over-parameterization bound m, we employ several tight analysis techniques from previous studies [Song and Yang arXiv 2019, Munteanu, Omlor, Song and Woodruff ICML 2022].