# Skewed Neuronal Heterogeneity Enhances Efficiency On Various Computing Systems

arXiv: https://arxiv.org/abs/2412.05126 · HTML: https://mlsanity.com/paper/2412.05126/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 142 references that are in the map.
- First posted on arXiv 6 Dec 2024.

## Builds on (most cited first, 30 of 33)

- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks](https://mlsanity.com/paper/2007.05785/) (2020)
- [Learning Factored Representations in a Deep Mixture of Experts](https://mlsanity.com/paper/1312.4314/) (2013)
- [Inductive Biases for Deep Learning of Higher-Level Cognition](https://mlsanity.com/paper/2011.15091/) (2020)
- [Maxout Networks](https://mlsanity.com/paper/1302.4389/) (2013)
- [Training Deep Neural Networks with 8-bit Floating Point Numbers](https://mlsanity.com/paper/1812.08011/) (2018)
- [Chaos as an interpretable benchmark for forecasting and data-driven modelling](https://mlsanity.com/paper/2110.05266/) (2021)
- [Echo state networks are universal](https://mlsanity.com/paper/1806.00797/) (2018)
- [Datasets for Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2402.18041/) (2024)
- [Towards Biologically Plausible Deep Learning](https://mlsanity.com/paper/1502.04156/) (2015)
- [Approximation Bounds for Random Neural Networks and Reservoir Systems](https://mlsanity.com/paper/2002.05933/) (2020)
- [Brain-inspired computing: We need a master plan](https://mlsanity.com/paper/2104.14517/) (2021)
- [Efficient Video and Audio processing with Loihi 2](https://mlsanity.com/paper/2310.03251/) (2023)
- [NeuroBench: A Framework for Benchmarking Neuromorphic Computing Algorithms and Systems](https://mlsanity.com/paper/2304.04640/) (2023)
- [Artificial neural networks for neuroscientists: A primer](https://mlsanity.com/paper/2006.01001/) (2020)
- [Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy](https://mlsanity.com/paper/2312.10549/) (2023)
- [A Unifying Framework for Information Processing in Stochastically Driven Dynamical Systems](https://mlsanity.com/paper/1906.04608/) (2019)
- [The SpiNNaker 2 Processing Element Architecture for Hybrid Digital Neuromorphic Computing](https://mlsanity.com/paper/2103.08392/) (2021)
- [Meta-Learning Biologically Plausible Plasticity Rules with Random Feedback Pathways](https://mlsanity.com/paper/2210.16414/) (2022)
- [Reservoir Computing with Diverse Timescales for Prediction of Multiscale Dynamics](https://mlsanity.com/paper/2108.09446/) (2021)
- [What is the dynamical regime of cerebral cortex?](https://mlsanity.com/paper/1908.10101/) (2019)
- [Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks](https://mlsanity.com/paper/2109.15089/) (2021)
- [Neuromorphic hardware for sustainable AI data centers](https://mlsanity.com/paper/2402.02521/) (2024)
- [A Review of Neuroscience-Inspired Machine Learning](https://mlsanity.com/paper/2403.18929/) (2024)
- [ON-OFF Neuromorphic ISING Machines using Fowler-Nordheim Annealers](https://mlsanity.com/paper/2406.05224/) (2024)
- [Adapting to time: Why nature may have evolved a diverse set of neurons](https://mlsanity.com/paper/2404.14325/) (2024)
- [Learning in High-Dimensional Multimedia Data: The State of the Art](https://mlsanity.com/paper/1707.02683/) (2017)
- [Optimal responsiveness and collective oscillations emerging from the heterogeneity of inhibitory neurons](https://mlsanity.com/paper/2005.05596/) (2020)
- [An accurate and flexible analog emulation of AdEx neuron dynamics in silicon](https://mlsanity.com/paper/2209.09280/) (2022)
- [Low-Precision Floating-Point for Efficient On-Board Deep Neural Network Processing](https://mlsanity.com/paper/2311.11172/) (2023)

## Abstract (arXiv)

Heterogeneity is a ubiquitous property of many biological systems and has profound implications for computation. While it is conceivable to optimize neuronal and synaptic heterogeneity for a specific task, such top-down optimization is biologically implausible, prone to catastrophic forgetting, and both data- and energy-intensive. In contrast, biological organisms, with remarkable capacity to perform numerous tasks with minimal metabolic cost, exhibit a heterogeneity that is inherent, stable during adulthood, and task-unspecific. Inspired by this intrinsic form of heterogeneity, we investigate the utility of variations in neuronal time constants for solving hundreds of distinct temporal tasks of varying complexity. Our results show that intrinsic heterogeneity significantly enhances performance and robustness in an implementation-independent manner, indicating its usefulness for both (rate-based) machine learning and (spike-coded) neuromorphic applications. Importantly, only skewed heterogeneity profiles-reminiscent of those found in biology-produce such performance gains. We further demonstrate that this computational advantage eliminates the need for large networks, allowing comparable performance with substantially lower operational, metabolic, and energetic costs, respectively in silico, in vivo, and on neuromorphic hardware. Finally, we discuss the implications of intrinsic (rather than task-induced) heterogeneity for the design of efficient artificial systems, particularly novel neuromorphic devices that exhibit similar device-to-device variability.