# How (Implicit) Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part II: the Multi-D Case of Two Layers with Random First Layer

arXiv: https://arxiv.org/abs/2303.11454 · HTML: https://mlsanity.com/paper/2303.11454/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 21 references that are in the map.
- First posted on arXiv 20 Mar 2023.

## Builds on (most cited first, 6 of 6)

- [Implicit Regularization of Random Feature Models](https://mlsanity.com/paper/2002.08404/) (2020)
- [A generative adversarial network approach to calibration of local stochastic volatility models](https://mlsanity.com/paper/2005.02505/) (2020)
- [Optimal Stopping via Randomized Neural Networks](https://mlsanity.com/paper/2104.13669/) (2021)
- [NOMU: Neural Optimization-based Model Uncertainty](https://mlsanity.com/paper/2102.13640/) (2021)
- [Implicit Bias of Gradient Descent for Mean Squared Error Regression with Two-Layer Wide Neural Networks](https://mlsanity.com/paper/2006.07356/) (2020)
- [Bayesian Optimization-based Combinatorial Assignment](https://mlsanity.com/paper/2208.14698/) (2022)

## Built on by (most cited first, 3 of 3)

- [Extending Path-Dependent NJ-ODEs to Noisy Observations and a Dependent Observation Framework](https://mlsanity.com/paper/2307.13147/) (2023)
- [Prices, Bids, Values: One ML-Powered Combinatorial Auction to Rule Them All](https://mlsanity.com/paper/2411.09355/) (2024)
- [Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse](https://mlsanity.com/paper/2607.03613/) (2026)

## Abstract (arXiv)

Randomized neural networks (randomized NNs), where only the terminal layer's weights are optimized constitute a powerful model class to reduce computational time in training the neural network model. At the same time, these models generalize surprisingly well in various regression and classification tasks. In this paper, we give an exact macroscopic characterization (i.e., a characterization in function space) of the generalization behavior of randomized, shallow NNs with ReLU activation (RSNs). We show that RSNs correspond to a generalized additive model (GAM)-typed regression in which infinitely many directions are considered: the infinite generalized additive model (IGAM). The IGAM is formalized as solution to an optimization problem in function space for a specific regularization functional and a fairly general loss. This work is an extension to multivariate NNs of prior work, where we showed how wide RSNs with ReLU activation behave like spline regression under certain conditions and if the input is one-dimensional.