# On the choice of the non-trainable internal weights in random feature maps

arXiv: https://arxiv.org/abs/2408.03626 · HTML: https://mlsanity.com/paper/2408.03626/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 50 references that are in the map.
- First posted on arXiv 7 Aug 2024.

## Builds on (most cited first, 18 of 18)

- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network](https://mlsanity.com/paper/1808.03314/) (2018)
- [Next Generation Reservoir Computing](https://mlsanity.com/paper/2106.07688/) (2021)
- [Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond](https://mlsanity.com/paper/2004.11154/) (2020)
- [Understanding LSTM -- a tutorial into Long Short-Term Memory Recurrent Neural Networks](https://mlsanity.com/paper/1909.09586/) (2019)
- [Regularization for Deep Learning: A Taxonomy](https://mlsanity.com/paper/1710.10686/) (2017)
- [The Random Feature Model for Input-Output Maps between Banach Spaces](https://mlsanity.com/paper/2005.10224/) (2020)
- [Echo state networks are universal](https://mlsanity.com/paper/1806.00797/) (2018)
- [A Framework for Machine Learning of Model Error in Dynamical Systems](https://mlsanity.com/paper/2107.06658/) (2021)
- [Fading memory echo state networks are universal](https://mlsanity.com/paper/2010.12047/) (2020)
- [Supervised learning from noisy observations: Combining machine-learning techniques with data assimilation](https://mlsanity.com/paper/2007.07383/) (2020)
- [EA-LSTM: Evolutionary Attention-based LSTM for Time Series Prediction](https://mlsanity.com/paper/1811.03760/) (2018)
- [Combining machine learning and data assimilation to forecast dynamical systems from noisy partial observations](https://mlsanity.com/paper/2108.03561/) (2021)
- [Convergence of Gibbs Sampling: Coordinate Hit-and-Run Mixes Fast](https://mlsanity.com/paper/2009.11338/) (2020)
- [Variational training of neural network approximations of solution maps for physical models](https://mlsanity.com/paper/1905.02789/) (2019)
- [Hyperparameter Optimization for Randomized Algorithms: A Case Study on Random Features](https://mlsanity.com/paper/2407.00584/) (2024)
- [Hit-and-Run for Sampling and Planning in Non-Convex Spaces](https://mlsanity.com/paper/1610.08865/) (2016)

## Built on by (most cited first, 2 of 2)

- [Learning dynamical systems with hit-and-run random feature maps](https://mlsanity.com/paper/2501.06661/) (2025)
- [Learning Chaotic Dynamics with Neuromorphic Network Dynamics](https://mlsanity.com/paper/2506.10773/) (2025)

## Abstract (arXiv)

The computationally cheap machine learning architecture of random feature maps can be viewed as a single-layer feedforward network in which the weights of the hidden layer are random but fixed and only the outer weights are learned via linear regression. The internal weights are typically chosen from a prescribed distribution. The choice of the internal weights significantly impacts the accuracy of random feature maps. We address here the task of how to best select the internal weights. In particular, we consider the forecasting problem whereby random feature maps are used to learn a one-step propagator map for a dynamical system. We provide a computationally cheap hit-and-run algorithm to select good internal weights which lead to good forecasting skill. We show that the number of good features is the main factor controlling the forecasting skill of random feature maps and acts as an effective feature dimension. Lastly, we compare random feature maps with single-layer feedforward neural networks in which the internal weights are now learned using gradient descent. We find that random feature maps have superior forecasting capabilities whilst having several orders of magnitude lower computational cost.