mlsanityData to 14 Oct 2026

Skewed Neuronal Heterogeneity Enhances Efficiency On Various Computing Systems

arXiv 2412.05126 ↗First posted 6 Dec 2024cs.LG cs.NEcited by 0

Not on the map. Why?

Abstract (arXiv)

Heterogeneity is a ubiquitous property of many biological systems and has profound implications for computation. While it is conceivable to optimize neuronal and synaptic heterogeneity for a specific task, such top-down optimization is biologically implausible, prone to catastrophic forgetting, and both data- and energy-intensive. In contrast, biological organisms, with remarkable capacity to perform numerous tasks with minimal metabolic cost, exhibit a heterogeneity that is inherent, stable during adulthood, and task-unspecific. Inspired by this intrinsic form of heterogeneity, we investigate the utility of variations in neuronal time constants for solving hundreds of distinct temporal tasks of varying complexity. Our results show that intrinsic heterogeneity significantly enhances performance and robustness in an implementation-independent manner, indicating its usefulness for both (rate-based) machine learning and (spike-coded) neuromorphic applications. Importantly, only skewed heterogeneity profiles-reminiscent of those found in biology-produce such performance gains. We further demonstrate that this computational advantage eliminates the need for large networks, allowing comparable performance with substantially lower operational, metabolic, and energetic costs, respectively in silico, in vivo, and on neuromorphic hardware. Finally, we discuss the implications of intrinsic (rather than task-induced) heterogeneity for the design of efficient artificial systems, particularly novel neuromorphic devices that exhibit similar device-to-device variability.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 142 references that are in the map. First posted on arXiv 6 Dec 2024.

Builds on

33 of 142 references.

References · 33

  1. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  2. 2020Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networkscited by 345
  3. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
  4. 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 275
  5. 2013Maxout Networkscited by 187
  6. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 164
  7. 2021Chaos as an interpretable benchmark for forecasting and data-driven modellingcited by 88
  8. 2018Echo state networks are universalcited by 87
  9. 2024Datasets for Large Language Models: A Comprehensive Surveycited by 71
  10. 2015Towards Biologically Plausible Deep Learningcited by 66
  11. 2020Approximation Bounds for Random Neural Networks and Reservoir Systemscited by 52
  12. 2021Brain-inspired computing: We need a master plancited by 51
21 more
  1. 2023Efficient Video and Audio processing with Loihi 2cited by 29
  2. 2023NeuroBench: A Framework for Benchmarking Neuromorphic Computing Algorithms and Systemscited by 27
  3. 2020Artificial neural networks for neuroscientists: A primercited by 26
  4. 2023Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomycited by 24
  5. 2019A Unifying Framework for Information Processing in Stochastically Driven Dynamical Systemscited by 16
  6. 2021The SpiNNaker 2 Processing Element Architecture for Hybrid Digital Neuromorphic Computingcited by 14
  7. 2022Meta-Learning Biologically Plausible Plasticity Rules with Random Feedback Pathwayscited by 14
  8. 2021Reservoir Computing with Diverse Timescales for Prediction of Multiscale Dynamicscited by 13
  9. 2019What is the dynamical regime of cerebral cortex?cited by 6
  10. 2021Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networkscited by 6
  11. 2024Neuromorphic hardware for sustainable AI data centerscited by 6
  12. 2024A Review of Neuroscience-Inspired Machine Learningcited by 6
  13. 2024ON-OFF Neuromorphic ISING Machines using Fowler-Nordheim Annealerscited by 6
  14. 2024Adapting to time: Why nature may have evolved a diverse set of neuronscited by 5
  15. 2017Learning in High-Dimensional Multimedia Data: The State of the Artcited by 4
  16. 2020Optimal responsiveness and collective oscillations emerging from the heterogeneity of inhibitory neuronscited by 3
  17. 2022An accurate and flexible analog emulation of AdEx neuron dynamics in siliconcited by 3
  18. 2023Low-Precision Floating-Point for Efficient On-Board Deep Neural Network Processingcited by 3
  19. 2016Death and rebirth of neural activity in sparse inhibitory networkscited by 1
  20. 2019Neural Activity of Heterogeneous Inhibitory Spiking Networks with Delaycited by 1
  21. 2024Plastic Arbor: a modern simulation framework for synaptic plasticity -- from single synapses to networks of morphological neuronscited by 1

Built on by

Not cited yet.