# Interneurons accelerate learning dynamics in recurrent neural networks for statistical adaptation

arXiv: https://arxiv.org/abs/2209.10634 · HTML: https://mlsanity.com/paper/2209.10634/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 52 references that are in the map.
- First posted on arXiv 21 Sep 2022.

## Builds on (most cited first, 17 of 17)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization](https://mlsanity.com/paper/1802.06509/) (2018)
- [Whitening for Self-Supervised Representation Learning](https://mlsanity.com/paper/2007.06346/) (2020)
- [A mathematical theory of semantic development in deep neural networks](https://mlsanity.com/paper/1810.10531/) (2018)
- [On Feature Decorrelation in Self-Supervised Learning](https://mlsanity.com/paper/2105.00470/) (2021)
- [Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks](https://mlsanity.com/paper/1904.13262/) (2019)
- [Optimal whitening and decorrelation](https://mlsanity.com/paper/1512.00809/) (2015)
- [Decorrelated Batch Normalization](https://mlsanity.com/paper/1804.08450/) (2018)
- [Why do similarity matching objectives lead to Hebbian/anti-Hebbian networks?](https://mlsanity.com/paper/1703.07914/) (2017)
- [A Normative Theory of Adaptive Dimensionality Reduction in Neural Networks](https://mlsanity.com/paper/1511.09426/) (2015)
- [Blind nonnegative source separation using biological neural networks](https://mlsanity.com/paper/1706.00382/) (2017)
- [A biologically plausible neural network for multi-channel Canonical Correlation Analysis](https://mlsanity.com/paper/2010.00525/) (2020)
- [Biologically plausible single-layer networks for nonnegative independent component analysis](https://mlsanity.com/paper/2010.12632/) (2020)
- [A simple normative network approximates local non-Hebbian learning in the cortex](https://mlsanity.com/paper/2010.12660/) (2020)

## Built on by (most cited first, 4 of 4)

- [Normative framework for deriving neural networks with multi-compartmental neurons and non-Hebbian plasticity](https://mlsanity.com/paper/2302.10051/) (2023)
- [Adaptive whitening in neural populations with gain-modulating interneurons](https://mlsanity.com/paper/2301.11955/) (2023)
- [Adaptive whitening with fast gain modulation and slow synaptic plasticity](https://mlsanity.com/paper/2308.13633/) (2023)
- [Discretization of continuous input spaces in the hippocampal autoencoder](https://mlsanity.com/paper/2405.14600/) (2024)

## Abstract (arXiv)

Early sensory systems in the brain rapidly adapt to fluctuating input statistics, which requires recurrent communication between neurons. Mechanistically, such recurrent communication is often indirect and mediated by local interneurons. In this work, we explore the computational benefits of mediating recurrent communication via interneurons compared with direct recurrent connections. To this end, we consider two mathematically tractable recurrent linear neural networks that statistically whiten their inputs -- one with direct recurrent connections and the other with interneurons that mediate recurrent communication. By analyzing the corresponding continuous synaptic dynamics and numerically simulating the networks, we show that the network with interneurons is more robust to initialization than the network with direct recurrent connections in the sense that the convergence time for the synaptic dynamics in the network with interneurons (resp. direct recurrent connections) scales logarithmically (resp. linearly) with the spectrum of their initialization. Our results suggest that interneurons are computationally useful for rapid adaptation to changing input statistics. Interestingly, the network with interneurons is an overparameterized solution of the whitening objective for the network with direct recurrent connections, so our results can be viewed as a recurrent linear neural network analogue of the implicit acceleration phenomenon observed in overparameterized feedforward linear neural networks.