mlsanityData to 14 Oct 2026

Entangled Residual Mappings

arXiv 2206.01261 ↗First posted 2 Jun 2022cs.LG cs.AI cs.NEcited by 4

Not on the map. Why?

Abstract (arXiv)

Residual mappings have been shown to perform representation learning in the first layers and iterative feature refinement in higher layers. This interplay, combined with their stabilizing effect on the gradient norms, enables them to train very deep networks. In this paper, we take a step further and introduce entangled residual mappings to generalize the structure of the residual connections and evaluate their role in iterative learning representations. An entangled residual mapping replaces the identity skip connections with specialized entangled mappings such as orthogonal, sparse, and structural correlation matrices that share key attributes (eigenvalues, structure, and Jacobian norm) with identity mappings. We show that while entangled mappings can preserve the iterative refinement of features across various deep models, they influence the representation learning process in convolutional networks differently than attention-based models and recurrent neural networks. In general, we find that for CNNs and Vision Transformers entangled sparse mapping can help generalization while orthogonal mappings hurt performance. For recurrent networks, orthogonal residual mappings form an inductive bias for time-variant sequences, which degrades accuracy on time-invariant tasks.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 57 of its 82 references that are in the map. First posted on arXiv 2 Jun 2022.

Builds on

57 of 82 references.

References · 57

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  7. 2016Densely Connected Convolutional Networkscited by 6,720
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  9. 2018Neural Ordinary Differential Equationscited by 3,779
  10. 2016Wide Residual Networkscited by 3,070
  11. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  12. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
38 more
  1. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  2. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  3. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,512
  4. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  5. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  6. 2018FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Modelscited by 594
  7. 2018Invertible Residual Networkscited by 365
  8. 2015LSTM: A Search Space Odysseycited by 316
  9. 2015Unitary Evolution Recurrent Neural Networkscited by 294
  10. 2015Highway Networkscited by 260
  11. 2019Residual Flows for Invertible Generative Modelingcited by 257
  12. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
  13. 2015Training Very Deep Networkscited by 173
  14. 2020Liquid Time-constant Networkscited by 161
  15. 2017The Shattered Gradients Problem: If resnets are the answer, then what is the question?cited by 157
  16. 2020Dissecting Neural ODEscited by 153
  17. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 134
  18. 2015All you need is a good initcited by 124
  19. 2019Orthogonal Convolutional Neural Networkscited by 123
  20. 2017Dual Path Networkscited by 119
  21. 2016Full-Capacity Unitary Recurrent Neural Networkscited by 94
  22. 2017Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networkscited by 92
  23. 2017On orthogonality and learning recurrent networks with long term dependenciescited by 83
  24. 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
  25. 2020Learning Long-Term Dependencies in Irregularly-Sampled Time Seriescited by 83
  26. 2017Neural Optimizer Search with Reinforcement Learningcited by 82
  27. 2017Residual Connections Encourage Iterative Inferencecited by 77
  28. 2016Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNscited by 65
  29. 2018Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?cited by 62
  30. 2017All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulationcited by 58
  31. 2016Highway and Residual Networks learn Unrolled Iterative Estimationcited by 52
  32. 2019Cascaded Context Pyramid for Full-Resolution 3D Semantic Scene Completioncited by 43
  33. 2016Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortexcited by 42
  34. 2016Regularizing CNNs with Locally Constrained Decorrelationscited by 41
  35. 2016Recurrent Highway Networkscited by 37
  36. 2019How to Initialize your Network? Robust Initialization for WeightNorm & ResNetscited by 36
  37. 2021Causal Navigation by Continuous-time Neural Networkscited by 23
  38. 2018Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Functioncited by 22

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 0; 2024: 0; 2025: 1.

4 papers.

  1. 2022Evolution of Neural Tangent Kernels under Benign and Adversarial Trainingcited by 14
  2. 2022Are All Vision Models Created Equal? A Study of the Open-Loop to Closed-Loop Causality Gapcited by 5
  3. 2025Revisiting Residual Connections: Orthogonal Updates for Stable and Efficient Deep Networkscited by 2
  4. 2022Interpreting Neural Policies with Disentangled Tree Representationscited by 1