mlsanityData to 7 Oct 2026

Residual Connections Encourage Iterative Inference

arXiv 1710.04773 ↗First posted 13 Oct 2017cs.CVcited by 74

Not on the map. Why?

Abstract (arXiv)

Residual networks (Resnets) have become a prominent architecture in deep learning. However, a comprehensive understanding of Resnets is still a topic of ongoing research. A recent view argues that Resnets perform iterative refinement of features. We attempt to further expose properties of this aspect. To this end, we study Resnets both analytically and empirically. We formalize the notion of iterative refinement in Resnets by showing that residual connections naturally encourage features of residual blocks to move along the negative gradient of loss as we go from one block to the next. In addition, our empirical analysis suggests that Resnets are able to perform both representation learning and iterative refinement. In general, a Resnet block tends to concentrate representation learning behavior in the first few layers while higher layers perform iterative refinement of features. Finally we observe that sharing residual layers naively leads to representation explosion and counterintuitively, overfitting, and we show that simple existing strategies can help alleviating this problem.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Oct 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 12; 2023: 11; 2024: 22; 2025: 13; 2026: 12.

74 papers.

Sort:

By area:

  1. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  2. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  3. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  4. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  5. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  6. 2024Universal Neurons in GPT2 Language Modelscited by 100
  7. 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 97
  8. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  9. 2024Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?cited by 60
  10. 2024Emergence of a High-Dimensional Abstraction Phase in Language Transformerscited by 56
  11. 2022Architecture Matters in Continual Learningcited by 46
  12. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
38 more
  1. 2024Explorations of Self-Repair in Language Modelscited by 35
  2. 2024Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questionscited by 35
  3. 2025Shared Global and Local Geometry of Language Model Embeddingscited by 34
  4. 2024Calibrating Reasoning in Language Models with Internal Consistencycited by 27
  5. 2023DecoderLens: Layerwise Interpretation of Encoder-Decoder Transformerscited by 22
  6. 2024Residual Stream Analysis with Multi-Layer SAEscited by 18
  7. 2025The Intrinsic Dimension of Prompts in Internal Representations of Large Language Modelscited by 17
  8. 2022Lambda-DARTS: Mitigating Performance Collapse by Harmonizing Operation Selection among Cellscited by 15
  9. 2023SR-init: An interpretable layer pruning methodcited by 13
  10. 2025LinEAS: End-to-end Learning of Activation Steering with a Distributional Losscited by 12
  11. 2021The Uncanny Similarity of Recurrence and Depthcited by 9
  12. 2022Parameter-Efficient Masking Networkscited by 9
  13. 2024Does Transformer Interpretability Transfer to RNNs?cited by 9
  14. 2024Transformer Block Coupling and its Correlation with Generalization in LLMscited by 9
  15. 2024Rethinking Deep Thinking: Stable Learning of Algorithms using Lipschitz Constraintscited by 8
  16. 2025Understanding Multi-View Transformerscited by 8
  17. 2023Traveling Words: A Geometric Interpretation of Transformerscited by 7
  18. 2022Learning Features with Parameter-Free Layerscited by 5
  19. 2023Power-Enhanced Residual Network for Function Approximation and Physics-Informed Inverse Problemscited by 5
  20. 2022Entangled Residual Mappingscited by 4
  21. 2023Module-wise Training of Neural Networks via the Minimizing Movement Schemecited by 4
  22. 2024Stable Weight Updating: A Key to Reliable PDE Solutions Using Deep Learningcited by 4
  23. 2024Jet Expansions of Residual Computationcited by 4
  24. 2025XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerializationcited by 4
  25. 2026Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Modelscited by 4
  26. 2024Understanding the role of FFNs in driving multilingual behaviour in LLMscited by 3
  27. 2024Training the Untrainable: Introducing Inductive Bias via Representational Alignmentcited by 3
  28. 2021Pruning Edges and Gradients to Learn Hypergraphs from Larger Setscited by 2
  29. 2021Category-orthogonal object features guide information processing in recurrent neural networks trained for object categorizationcited by 2
  30. 2021Hidden-Fold Networks: Random Recurrent Residuals Using Sparse Supermaskscited by 2
  31. 2023DTP-Net: Learning to Reconstruct EEG signals in Time-Frequency Domain by Multi-scale Feature Reusecited by 2
  32. 2022End-to-End Sensitivity-Based Filter Pruningcited by 1
  33. 2022SPIN: An Empirical Evaluation on Sharing Parameters of Isotropic Networkscited by 1
  34. 2022Transfer Learning and Vision Transformer based State-of-Health prediction of Lithium-Ion Batteriescited by 1
  35. 2022Block-wise Training of Residual Networks via the Minimizing Movement Schemecited by 1
  36. 2022Vision Transformer Computation and Resilience for Dynamic Inferencecited by 1
  37. 2023Adaptive Depth Networks with Skippable Sub-Pathscited by 1
  38. 2024Provable optimal transport with transformers: The essence of depth and prompt engineeringcited by 1

50 most cited of 74 All 74 (JSON)