Approximation with SiLU Networks: Constant Depth and Exponential Rates for Basic Operations

We present SiLU network constructions whose approximation efficiency depends critically on proper hyperparameter tuning. For the square function x2, with optimally chosen shift a and scale β, we achieve approximation error varepsilon using a two-layer network of constant width, where weights scale as β± k with k = O(ln(1/varepsilon)). We then extend this approach through functional composition to Sobolev spaces, we obtain networks with depth O(1) and O(varepsilon-d/n) parameters under optimal hyperparameters settings. Our work highlights the trade-off between architectural depth and activation parameter optimization in neural network approximation theory.