mlsanityData to 14 Oct 2026

Deep equilibrium networks are sensitive to initialization statistics

arXiv 2207.09432 ↗First posted 19 Jul 2022cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Deep equilibrium networks (DEQs) are a promising way to construct models which trade off memory for compute. However, theoretical understanding of these models is still lacking compared to traditional networks, in part because of the repeated application of a single set of weights. We show that DEQs are sensitive to the higher order statistics of the matrix families from which they are initialized. In particular, initializing with orthogonal or symmetric matrices allows for greater stability in training. This gives us a practical prescription for initializations which allow for training with a broader range of initial weight scales.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 26 of its 35 references that are in the map. First posted on arXiv 19 Jul 2022.

Builds on

26 of 35 references.

References · 26

  1. 2018Neural Ordinary Differential Equationscited by 3,779
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  5. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  6. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  7. 2019Deep Equilibrium Modelscited by 615
  8. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  9. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  10. 2020Dissecting Neural ODEscited by 153
  11. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 151
  12. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
14 more
  1. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  2. 2021Deep Equilibrium Architectures for Inverse Problems in Imagingcited by 127
  3. 2019Implicit Deep Learningcited by 124
  4. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 115
  5. 2019AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networkscited by 112
  6. 2020Monotone operator equilibrium networkscited by 100
  7. 2020Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networkscited by 73
  8. 2021Stabilizing Equilibrium Models by Jacobian Regularizationcited by 72
  9. 2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 52
  10. 2020Lipschitz Bounded Equilibrium Networkscited by 49
  11. 2021On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layerscited by 35
  12. 2020On the Neural Tangent Kernel of Deep Networks with Orthogonal Initializationcited by 21
  13. 2021A global convergence theory for deep ReLU implicit networks via over-parameterizationcited by 17
  14. 2021On the validity of kernel approximations for orthogonally-initialized neural networkscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 1; 2026: 3.

8 papers.

  1. 2024Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixturescited by 6
  2. 2024Towards training digitally-tied analog blocks via hybrid gradient computationcited by 4
  3. 2023iMixer: hierarchical Hopfield network implies an invertible, implicit and iterative MLP-Mixercited by 3
  4. 2026Derivative Informed Learning of Exchange-Correlation Functionalscited by 2
  5. 2026Lipschitz Multiscale Deep Equilibrium Models: A Theoretically Guaranteed and Accelerated Approachcited by 1
  6. 2026Response Renormalization for Critical Deep Equilibrium Modelscited by 1
  7. 2023Revisiting Implicit Models: Sparsity Trade-offs Capability in Weight-tied Model for Vision Taskscited by 0
  8. 2025On The Finetuning of MLIPs Through the Lens of Iterated Maps With BPTTcited by 0