# Sparsity through evolutionary pruning prevents neuronal networks from overfitting

arXiv: https://arxiv.org/abs/1911.10988 · HTML: https://mlsanity.com/paper/1911.10988/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2019.

## Built on by (most cited first, 11 of 11)

- [Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm](https://mlsanity.com/paper/2110.08190/) (2021)
- [Tuning-Free Accountable Intervention for LLM Deployment -- A Metacognitive Approach](https://mlsanity.com/paper/2403.05636/) (2024)
- [Neural Network based Successor Representations of Space and Language](https://mlsanity.com/paper/2202.11190/) (2022)
- [Word class representations spontaneously emerge in a deep neural network trained on next word prediction](https://mlsanity.com/paper/2302.07588/) (2023)
- [Fighting over-fitting with quantization for learning deep neural networks on noisy labels](https://mlsanity.com/paper/2303.11803/) (2023)
- [Always-Sparse Training by Growing Connections with Guided Stochastic Exploration](https://mlsanity.com/paper/2401.06898/) (2024)
- [Analyzing Narrative Processing in Large Language Models (LLMs): Using GPT4 to test BERT](https://mlsanity.com/paper/2405.02024/) (2024)
- [Predictive coding and stochastic resonance as fundamental principles of auditory perception](https://mlsanity.com/paper/2204.03354/) (2022)
- [EPIC TTS Models: Empirical Pruning Investigations Characterizing Text-To-Speech Models](https://mlsanity.com/paper/2209.10890/) (2022)
- [On Diversity in Discriminative Neural Networks](https://mlsanity.com/paper/2407.12599/) (2024)
- [Sparsity-Aware Hardware-Software Co-Design of Spiking Neural Networks: An Overview](https://mlsanity.com/paper/2408.14437/) (2024)

## Abstract (arXiv)

Modern Machine learning techniques take advantage of the exponentially rising calculation power in new generation processor units. Thus, the number of parameters which are trained to resolve complex tasks was highly increased over the last decades. However, still the networks fail - in contrast to our brain - to develop general intelligence in the sense of being able to solve several complex tasks with only one network architecture. This could be the case because the brain is not a randomly initialized neural network, which has to be trained by simply investing a lot of calculation power, but has from birth some fixed hierarchical structure. To make progress in decoding the structural basis of biological neural networks we here chose a bottom-up approach, where we evolutionarily trained small neural networks in performing a maze task. This simple maze task requires dynamical decision making with delayed rewards. We were able to show that during the evolutionary optimization random severance of connections lead to better generalization performance of the networks compared to fully connected networks. We conclude that sparsity is a central property of neural networks and should be considered for modern Machine learning approaches.