# Expressivity of Neural Networks with Random Weights and Learned Biases

arXiv: https://arxiv.org/abs/2407.00957 · HTML: https://mlsanity.com/paper/2407.00957/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 78 references that are in the map.
- First posted on arXiv 1 Jul 2024.

## Builds on (most cited first, 30 of 30)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022)
- [Deep Learning for Classical Japanese Literature](https://mlsanity.com/paper/1812.01718/) (2018)
- [What's Hidden in a Randomly Weighted Neural Network?](https://mlsanity.com/paper/1911.13299/) (2019)
- [Proving the Lottery Ticket Hypothesis: Pruning is All You Need](https://mlsanity.com/paper/2002.00585/) (2020)
- [SuperSpike: Supervised learning in multi-layer spiking neural networks](https://mlsanity.com/paper/1705.11146/) (2017)
- [Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP](https://mlsanity.com/paper/1906.02768/) (2019)
- [Approximation Bounds for Random Neural Networks and Reservoir Systems](https://mlsanity.com/paper/2002.05933/) (2020)
- [Gated Linear Networks](https://mlsanity.com/paper/1910.01526/) (2019)
- [Prompting a Pretrained Transformer Can Be a Universal Approximator](https://mlsanity.com/paper/2402.14753/) (2024)
- [Echo State Networks trained by Tikhonov least squares are L2(μ) approximators of ergodic dynamical systems](https://mlsanity.com/paper/2005.06967/) (2020)
- [Intriguing Properties of Randomly Weighted Networks: Generalizing While Learning Next to Nothing](https://mlsanity.com/paper/1802.00844/) (2018)
- [Dynamical learning of dynamics](https://mlsanity.com/paper/1902.02875/) (2019)
- [Kannada-MNIST: A new handwritten digits dataset for the Kannada language](https://mlsanity.com/paper/1908.01242/) (2019)
- [The Expressive Power of Tuning Only the Normalization Layers](https://mlsanity.com/paper/2302.07937/) (2023)
- [Afro-MNIST: Synthetic generation of MNIST-style datasets for low-resource languages](https://mlsanity.com/paper/2009.13509/) (2020)
- [Metastable attractors explain the variable timing of stable behavioral action sequences](https://mlsanity.com/paper/2001.09600/) (2020)
- [Neural mechanisms underlying the temporal organization of naturalistic animal behavior](https://mlsanity.com/paper/2203.02151/) (2022)
- [How Powerful are Shallow Neural Networks with Bandlimited Random Weights?](https://mlsanity.com/paper/2008.08427/) (2020)
- [Expectation-induced modulation of metastable activity underlies faster coding of sensory stimuli](https://mlsanity.com/paper/1710.02563/) (2017)
- [State-dependent regulation of cortical processing speed via gain modulation](https://mlsanity.com/paper/2004.04190/) (2020)
- [Hidden-Fold Networks: Random Recurrent Residuals Using Sparse Supermasks](https://mlsanity.com/paper/2111.12330/) (2021)
- [Modulation of metastable ensemble dynamics explains the inverted-U relationship between tone discriminability and arousal in auditory cortex](https://mlsanity.com/paper/2404.03902/) (2024)
- [Macroscopic Dynamics of Neural Networks with Heterogeneous Spiking Thresholds](https://mlsanity.com/paper/2209.03501/) (2022)

## Built on by (most cited first, 2 of 2)

- [Improving the adaptive and continuous learning capabilities of artificial neural networks: Lessons from multi-neuromodulatory dynamics](https://mlsanity.com/paper/2501.06762/) (2025)
- [JEDI: Jointly Embedded Inference of Neural Dynamics](https://mlsanity.com/paper/2603.10489/) (2026)

## Abstract (arXiv)

Landmark universal function approximation results for neural networks with trained weights and biases provided the impetus for the ubiquitous use of neural networks as learning models in neuroscience and Artificial Intelligence (AI). Recent work has extended these results to networks in which a smaller subset of weights (e.g., output weights) are tuned, leaving other parameters random. However, it remains an open question whether universal approximation holds when only biases are learned, despite evidence from neuroscience and AI that biases significantly shape neural responses. The current paper answers this question. We provide theoretical and numerical evidence demonstrating that feedforward neural networks with fixed random weights can approximate any continuous function on compact sets. We further show an analogous result for the approximation of dynamical systems with recurrent neural networks. Our findings are relevant to neuroscience, where they demonstrate the potential for behaviourally relevant changes in dynamics without modifying synaptic weights, as well as for AI, where they shed light on recent fine-tuning methods for large language models, like bias and prefix-based approaches.