# On Computing the Hyperparameter of Extreme Learning Machines: Algorithm and Application to Computational PDEs, and Comparison with Classical and High-Order Finite Elements

arXiv: https://arxiv.org/abs/2110.14121 · HTML: https://mlsanity.com/paper/2110.14121/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 68 references that are in the map.
- First posted on arXiv 27 Oct 2021.

## Builds on (most cited first, 17 of 17)

- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [Variational Physics-Informed Neural Networks For Solving Partial Differential Equations](https://mlsanity.com/paper/1912.00873/) (2019)
- [An Energy Approach to the Solution of Partial Differential Equations in Computational Mechanics via Machine Learning: Concepts, Implementation and Applications](https://mlsanity.com/paper/1908.10407/) (2019)
- [A Method for Representing Periodic Functions and Enforcing Exactly Periodic Boundary Conditions with Deep Neural Networks](https://mlsanity.com/paper/2007.07442/) (2020)
- [Local Extreme Learning Machines and Domain Decomposition for Solving Linear and Nonlinear Partial Differential Equations](https://mlsanity.com/paper/2012.02895/) (2020)
- [Physics Informed Extreme Learning Machine (PIELM) -- A rapid method for the numerical solution of partial differential equations](https://mlsanity.com/paper/1907.03507/) (2019)
- [D3M: A deep domain decomposition method for partial differential equations](https://mlsanity.com/paper/1909.12236/) (2019)
- [Robust Training and Initialization of Deep Neural Networks: An Adaptive Basis Viewpoint](https://mlsanity.com/paper/1912.04862/) (2019)
- [Deep least-squares methods: an unsupervised learning-based numerical method for solving elliptic PDEs](https://mlsanity.com/paper/1911.02109/) (2019)
- [Adaptive deep density approximation for Fokker-Planck equations](https://mlsanity.com/paper/2103.11181/) (2021)
- [A Modified Batch Intrinsic Plasticity Method for Pre-training the Random Coefficients of Extreme Learning Machines](https://mlsanity.com/paper/2103.08042/) (2021)
- [VAE-KRnet and its applications to variational Bayes](https://mlsanity.com/paper/2006.16431/) (2020)
- [A Roadmap for Discretely Energy-Stable Schemes for Dissipative Systems Based on a Generalized Auxiliary Variable with Guaranteed Positivity](https://mlsanity.com/paper/1904.00141/) (2019)

## Built on by (most cited first, 26 of 26)

- [Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations](https://mlsanity.com/paper/2407.07218/) (2024)
- [An Extreme Learning Machine-Based Method for Computational PDEs in Higher Dimensions](https://mlsanity.com/paper/2309.07049/) (2023)
- [Parsimonious Physics-Informed Random Projection Neural Networks for Initial-Value Problems of ODEs and index-1 DAEs](https://mlsanity.com/paper/2203.05337/) (2022)
- [RandONet: Shallow-Networks with Random Projections for learning linear and nonlinear operators](https://mlsanity.com/paper/2406.05470/) (2024)
- [Data-driven Control of Agent-based Models: an Equation/Variable-free Machine Learning Approach](https://mlsanity.com/paper/2207.05779/) (2022)
- [A Method for Computing Inverse Parametric PDE Problems with Random-Weight Neural Networks](https://mlsanity.com/paper/2210.04338/) (2022)
- [Random Projection Neural Networks of Best Approximation: Convergence theory and practical applications](https://mlsanity.com/paper/2402.11397/) (2024)
- [Evolutionary Optimization of Physics-Informed Neural Networks: Evo-PINN Frontiers and Opportunities](https://mlsanity.com/paper/2501.06572/) (2025)
- [Slow Invariant Manifolds of Singularly Perturbed Systems via Physics-Informed Machine Learning](https://mlsanity.com/paper/2309.07946/) (2023)
- [Numerical Approximation of Partial Differential Equations by a Variable Projection Method with Artificial Neural Networks](https://mlsanity.com/paper/2201.09989/) (2022)
- [Numerical Computation of Partial Differential Equations by Hidden-Layer Concatenated Extreme Learning Machine](https://mlsanity.com/paper/2204.11375/) (2022)
- [Curriculum Learning-Driven PIELMs for Fluid Flow Simulations](https://mlsanity.com/paper/2503.06347/) (2025)
- [Kernel-Adaptive PI-ELMs for Forward and Inverse Problems in PDEs with Sharp Gradients](https://mlsanity.com/paper/2507.10241/) (2025)
- [Evolutionary Optimization of Physics-Informed Neural Networks: Advancing Generalizability by the Baldwin Effect](https://mlsanity.com/paper/2312.03243/) (2023)
- [Quantification of total uncertainty in the physics-informed reconstruction of CVSim-6 physiology](https://mlsanity.com/paper/2408.07201/) (2024)
- [Error Analysis of Physics-Informed Neural Networks for Approximating Dynamic PDEs of Second Order in Time](https://mlsanity.com/paper/2303.12245/) (2023)
- [Gated X-TFC: Soft Domain Decomposition for Forward and Inverse Problems in Sharp-Gradient PDEs](https://mlsanity.com/paper/2510.01039/) (2025)
- [A Novel Method for Enforcing Exactly Dirichlet, Neumann and Robin Conditions on Curved Domain Boundaries for Physics Informed Machine Learning](https://mlsanity.com/paper/2603.21909/) (2026)
- [On the accuracy of interpolation based on single-layer artificial neural networks with a focus on defeating the Runge phenomenon](https://mlsanity.com/paper/2308.10720/) (2023)
- [Error Analysis and Numerical Algorithm for PDE Approximation with Hidden-Layer Concatenated Physics Informed Neural Networks](https://mlsanity.com/paper/2406.06350/) (2024)
- [Fast PINN Eigensolvers via Biconvex Reformulation](https://mlsanity.com/paper/2511.00792/) (2025)
- [Exact Constraint Enforcement in Physics-Informed Extreme Learning Machines using Null-Space Projection Framework](https://mlsanity.com/paper/2601.10999/) (2026)
- [Stability and Bifurcation Analysis of Nonlinear PDEs via Random Projection-based PINNs: A Krylov-Arnoldi Approach](https://mlsanity.com/paper/2603.21568/) (2026)
- [Meta-Learned Basis Adaptation for Parametric Linear PDEs](https://mlsanity.com/paper/2604.09289/) (2026)
- [Multi-Scale Separable Fourier Neural Networks for Solving High-Frequency PDEs](https://mlsanity.com/paper/2605.31027/) (2026)
- [A Novel Fourier Feature Network for Solving Partial Differential Equations](https://mlsanity.com/paper/2608.14733/) (2026)

## Abstract (arXiv)

We consider the use of extreme learning machines (ELM) for computational partial differential equations (PDE). In ELM the hidden-layer coefficients in the neural network are assigned to random values generated on [-Rm,Rm] and fixed, where Rm is a user-provided constant, and the output-layer coefficients are trained by a linear or nonlinear least squares computation. We present a method for computing the optimal value of Rm based on the differential evolution algorithm. The presented method enables us to illuminate the characteristics of the optimal Rm for two types of ELM configurations: (i) Single-Rm-ELM, in which a single Rm is used for generating the random coefficients in all the hidden layers, and (ii) Multi-Rm-ELM, in which multiple Rm constants are involved with each used for generating the random coefficients of a different hidden layer. We adopt the optimal Rm from this method and also incorporate other improvements into the ELM implementation. In particular, here we compute all the differential operators involving the output fields of the last hidden layer by a forward-mode auto-differentiation, as opposed to the reverse-mode auto-differentiation in a previous work. These improvements significantly reduce the network training time and enhance the ELM performance. We systematically compare the computational performance of the current improved ELM with that of the finite element method (FEM), both the classical second-order FEM and the high-order FEM with Lagrange elements of higher degrees, for solving a number of linear and nonlinear PDEs. It is shown that the current improved ELM far outperforms the classical FEM. Its computational performance is comparable to that of the high-order FEM for smaller problem sizes, and for larger problem sizes the ELM markedly outperforms the high-order FEM.