mlsanityData to 7 Oct 2026

Multiscale Deep Equilibrium Models

arXiv 2006.08656 ↗First posted 15 Jun 2020cs.LG cs.CV stat.MLcited by 149

Not on the map. Why?

Abstract (arXiv)

We propose a new class of implicit networks, the multiscale deep equilibrium model (MDEQ), suited to large-scale and highly hierarchical pattern recognition domains. An MDEQ directly solves for and backpropagates through the equilibrium points of multiple feature resolutions simultaneously, using implicit differentiation to avoid storing intermediate states (and thus requiring only O(1) memory consumption). These simultaneously-learned multi-resolution features allow us to train a single model on a diverse set of tasks and loss functions, such as using a single MDEQ to perform both image classification and semantic segmentation. We illustrate the effectiveness of this approach on two large-scale vision tasks: ImageNet classification and semantic segmentation on high-resolution images from the Cityscapes dataset. In both settings, MDEQs are able to match or exceed the performance of recent competitive computer vision models: the first time such performance and scale have been achieved by an implicit deep learning approach. The code and pre-trained models are at https://github.com/locuslab/mdeq .

Key facts. Cited by 149 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 37; 2022: 34; 2023: 30; 2024: 19; 2025: 10; 2026: 19.

149 papers.

Sort:

By area:

  1. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  2. 2021Training Graph Neural Networks with 1000 Layerscited by 169
  3. 2021Neural SDEs as Infinite-Dimensional GANscited by 159
  4. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 135
  5. 2021You Only Learn One Representation: Unified Network for Multiple Taskscited by 86
  6. 2021JFB: Jacobian-Free Backpropagation for Implicit Networkscited by 84
  7. 2022Tutorial on amortized optimizationcited by 83
  8. 2021On Training Implicit Modelscited by 73
  9. 2021Stabilizing Equilibrium Models by Jacobian Regularizationcited by 71
  10. 2022Learning Deep Implicit Fourier Neural Operators (IFNOs) with Applications to Heterogeneous Material Modelingcited by 65
  11. 2022Deep Equilibrium Optical Flow Estimationcited by 63
  12. 2023One-Step Diffusion Distillation via Deep Equilibrium Modelscited by 60
38 more
  1. 2022Gradient Gating for Deep Multi-Rate Learning on Graphscited by 49
  2. 2021Vision Transformers with Patch Diversificationcited by 45
  3. 2022Nonlocal Kernel Network (NKN): a Stable and Resolution-Independent Deep Neural Networkcited by 45
  4. 2023Parallelizing non-linear sequential models over the sequence lengthcited by 45
  5. 2021CTNet: Context-based Tandem Network for Semantic Segmentationcited by 43
  6. 2022Deep Equilibrium Approaches to Diffusion Modelscited by 42
  7. 2021Long Expressive Memory for Sequence Modelingcited by 41
  8. 2021Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium Statecited by 39
  9. 2024Towards Scalable and Stable Parallelization of Nonlinear RNNscited by 38
  10. 2021If your data distribution shifts, use self-learningcited by 36
  11. 2022Path Independent Equilibrium Models Can Better Exploit Test-Time Computationcited by 33
  12. 2022EIGNN: Efficient Infinite-Depth Graph Neural Networkscited by 32
  13. 2022Reversible Column Networkscited by 32
  14. 2022Optimization-Induced Graph Implicit Nonlinear Diffusioncited by 28
  15. 2021Constraint-based graph network simulatorcited by 27
  16. 2021SHINE: SHaring the INverse Estimate from the forward pass for bi-level optimization and implicit modelscited by 26
  17. 2021Resolution learning in deep convolutional networks using scale-space theorycited by 26
  18. 2021Sliced Recursive Transformercited by 26
  19. 2022Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagationcited by 26
  20. 2023Recurrence without Recurrence: Stable Video Landmark Detection with Deep Equilibrium Modelscited by 26
  21. 2021Implicit differentiation for fast hyperparameter selection in non-smooth convex learningcited by 22
  22. 2022Alternating Differentiation for Optimization Layerscited by 22
  23. 2023TorchDEQ: A Library for Deep Equilibrium Modelscited by 21
  24. 2022MGNNI: Multiscale Graph Neural Networks with Implicit Layerscited by 19
  25. 2021Anytime Dense Prediction with Confidence Adaptivitycited by 18
  26. 2021Feasibility-based Fixed Point Networkscited by 17
  27. 2021A global convergence theory for deep ReLU implicit networks via over-parameterizationcited by 17
  28. 2021Multiset-Equivariant Set Prediction with Approximate Implicit Differentiationcited by 17
  29. 2022Bi-level Physics-Informed Neural Networks for PDE Constrained Optimization using Broyden's Hypergradientscited by 17
  30. 2023Deep Equilibrium Diffusion Restoration with Parallel Samplingcited by 17
  31. 2022The least-control principle for local learning at equilibriumcited by 16
  32. 2021Joint inference and input optimization in equilibrium networkscited by 15
  33. 2021A Unified Efficient Pyramid Transformer for Semantic Segmentationcited by 14
  34. 2022Explainable AI via Learning to Optimizecited by 14
  35. 2024Fixed Point Diffusion Modelscited by 14
  36. 2021Memory-Efficient Differentiable Transformer Architecture Searchcited by 13
  37. 2022Revisiting Sparse Convolutional Model for Visual Recognitioncited by 13
  38. 2023A Closer Look at the Adversarial Robustness of Deep Equilibrium Modelscited by 13

50 most cited of 149 All 149 (JSON)