# On the dynamics of convolutional recurrent neural networks near their critical point

arXiv: https://arxiv.org/abs/2405.13854 · HTML: https://mlsanity.com/paper/2405.13854/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 37 references that are in the map.
- First posted on arXiv 22 May 2024.

## Builds on (most cited first, 11 of 11)

- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Unitary Evolution Recurrent Neural Networks](https://mlsanity.com/paper/1511.06464/) (2015)
- [A Simple Way to Initialize Recurrent Networks of Rectified Linear Units](https://mlsanity.com/paper/1504.00941/) (2015)
- [AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks](https://mlsanity.com/paper/1902.09689/) (2019)
- [Recent Advances in Recurrent Neural Networks](https://mlsanity.com/paper/1801.01078/) (2017)
- [On orthogonality and learning recurrent networks with long term dependencies](https://mlsanity.com/paper/1702.00071/) (2017)
- [Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks](https://mlsanity.com/paper/2001.05992/) (2020)
- [Traveling Waves Encode the Recent Past and Enhance Sequence Learning](https://mlsanity.com/paper/2309.08045/) (2023)
- [Hidden Traveling Waves bind Working Memory Variables in Recurrent Neural Networks](https://mlsanity.com/paper/2402.10163/) (2024)
- [Convolutional unitary or orthogonal recurrent neural networks](https://mlsanity.com/paper/2302.07396/) (2023)
- [Adaptive Scales of Spatial Integration and Response Latencies in a Critically-Balanced Model of the Primary Visual Cortex](https://mlsanity.com/paper/1703.09347/) (2017)

## Built on by (most cited first, 1 of 1)

- [Input-driven circuit reconfiguration in critical recurrent neural networks](https://mlsanity.com/paper/2405.15036/) (2024)

## Abstract (arXiv)

We examine the dynamical properties of a single-layer convolutional recurrent network with a smooth sigmoidal activation function, for small values of the inputs and when the convolution kernel is unitary, so all eigenvalues lie exactly at the unit circle. Such networks have a variety of hallmark properties: the outputs depend on the inputs via compressive nonlinearities such as cubic roots, and both the timescales of relaxation and the length-scales of signal propagation depend sensitively on the inputs as power laws, both diverging as the input to 0. The basic dynamical mechanism is that inputs to the network generate ongoing activity, which in turn controls how additional inputs or signals propagate spatially or attenuate in time. We present analytical solutions for the steady states when the network is forced with a single oscillation and when a background value creates a steady state of ongoing activity, and derive the relationships shaping the value of the temporal decay and spatial propagation length as a function of this background value.