# A Study of Hybrid and Evolutionary Metaheuristics for Single Hidden Layer Feedforward Neural Network Architecture

arXiv: https://arxiv.org/abs/2506.15737 · HTML: https://mlsanity.com/paper/2506.15737/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 68 references that are in the map.
- First posted on arXiv 17 Jun 2025.

## Builds on (most cited first, 17 of 17)

- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Identifying and attacking the saddle point problem in high-dimensional non-convex optimization](https://mlsanity.com/paper/1406.2572/) (2014)
- [Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning](https://mlsanity.com/paper/1712.06567/) (2017)
- [A Literature Survey of Benchmark Functions For Global Optimization Problems](https://mlsanity.com/paper/1308.4008/) (2013)
- [Increasing Generality in Machine Learning through Procedural Content Generation](https://mlsanity.com/paper/1911.13071/) (2019)
- [Convolutional neural network classification of cancer cytopathology images: taking breast cancer as an example](https://mlsanity.com/paper/2404.08279/) (2024)
- [Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms](https://mlsanity.com/paper/2401.11963/) (2024)
- [Dealing with unbounded gradients in stochastic saddle-point optimization](https://mlsanity.com/paper/2402.13903/) (2024)
- [A Survey on Universal Approximation Theorems](https://mlsanity.com/paper/2407.12895/) (2024)
- [Convergence rates for the Adam optimizer](https://mlsanity.com/paper/2407.21078/) (2024)
- [A Survey on Activation Functions and their relation with Xavier and He Normal Initialization](https://mlsanity.com/paper/2004.06632/) (2020)
- [Coding schemes in neural networks learning classification tasks](https://mlsanity.com/paper/2406.16689/) (2024)
- [Best practices for comparing optimization algorithms](https://mlsanity.com/paper/1709.08242/) (2017)
- [Parameterless Gene-pool Optimal Mixing Evolutionary Algorithms](https://mlsanity.com/paper/2109.05259/) (2021)
- [Efficient Distributed Training through Gradient Compression with Sparsification and Quantization Techniques](https://mlsanity.com/paper/2502.07634/) (2024)
- [Bridging the Gap Between Theory and Practice: Benchmarking Transfer Evolutionary Optimization](https://mlsanity.com/paper/2404.13377/) (2024)

## Abstract (arXiv)

Training Artificial Neural Networks (ANNs) with Stochastic Gradient Descent (SGD) frequently encounters difficulties, including substantial computing expense and the risk of converging to local optima, attributable to its dependence on partial weight gradients. Therefore, this work investigates Particle Swarm Optimization (PSO) and Genetic Algorithms (GAs) - two population-based Metaheuristic Optimizers (MHOs) - as alternatives to SGD to mitigate these constraints. A hybrid PSO-SGD strategy is developed to improve local search efficiency. The findings indicate that the hybrid PSO-SGD technique decreases the median training MSE by 90 to 95 percent relative to conventional GA and PSO across various network sizes (e.g., from around 0.02 to approximately 0.001 in the Sphere function). RMHC attains substantial enhancements, reducing MSE by roughly 85 to 90 percent compared to GA. Simultaneously, RS consistently exhibits errors exceeding 0.3, signifying subpar performance. These findings underscore that hybrid and evolutionary procedures significantly improve training efficiency and accuracy compared to conventional optimization methods and imply that the Building Block Hypothesis (BBH) may still be valid, indicating that advantageous weight structures are retained during evolutionary search.