mlsanityData to 14 Oct 2026

Prospective Messaging: Learning in Networks with Communication Delays

arXiv 2407.05494 ↗First posted 7 Jul 2024cs.LG cs.NEcited by 0

Not on the map. Why?

Abstract (arXiv)

Inter-neuron communication delays are ubiquitous in physically realized neural networks such as biological neural circuits and neuromorphic hardware. These delays have significant and often disruptive consequences on network dynamics during training and inference. It is therefore essential that communication delays be accounted for, both in computational models of biological neural networks and in large-scale neuromorphic systems. Nonetheless, communication delays have yet to be comprehensively addressed in either domain. In this paper, we first show that delays prevent state-of-the-art continuous-time neural networks called Latent Equilibrium (LE) networks from learning even simple tasks despite significant overparameterization. We then propose to compensate for communication delays by predicting future signals based on currently available ones. This conceptually straightforward approach, which we call prospective messaging (PM), uses only neuron-local information, and is flexible in terms of memory and computation requirements. We demonstrate that incorporating PM into delayed LE networks prevents reaction lags, and facilitates successful learning on Fourier synthesis and autoregressive video prediction tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 87 references that are in the map. First posted on arXiv 7 Jul 2024.

Builds on

17 of 87 references.

References · 17

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  2. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  3. 2015Automatic differentiation in machine learning: a surveycited by 855
  4. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  5. 2016Decoupled Neural Interfaces using Synthetic Gradientscited by 137
  6. 2016Asynchronous Stochastic Gradient Descent with Delay Compensationcited by 99
  7. 2021Predictive Coding: a Theoretical and Experimental Reviewcited by 77
  8. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 76
  9. 2020A Modern Primer on Processing in Memorycited by 32
  10. 2021Latent Equilibrium: A unified learning theory for arbitrarily fast computation with arbitrarily slow neuronscited by 15
  11. 2015Deep Temporal Sigmoid Belief Networks for Sequence Modelingcited by 14
  12. 2022Axonal Delay As a Short-Term Memory for Feed Forward Deep Spiking Neural Networkscited by 11
5 more
  1. 2020Deep Neural Networks using a Single Neuron: Folded-in-Time Architecture using Feedback-Modulated Delay Loopscited by 9
  2. 2022DACOM: Learning Delay-Aware Communication for Multi-Agent Reinforcement Learningcited by 9
  3. 2020Optoelectronic Intelligencecited by 4
  4. 2013Delay Learning Architectures for Memory and Classificationcited by 2
  5. 2016Integration of continuous-time dynamics in a spiking neural network simulatorcited by 1

Built on by

Not cited yet.