mlsanityData to 14 Oct 2026

On the Origins of the Block Structure Phenomenon in Neural Network Representations

arXiv 2202.07184 ↗First posted 15 Feb 2022cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Recent work has uncovered a striking phenomenon in large-capacity neural networks: they contain blocks of contiguous hidden layers with highly similar representations. This block structure has two seemingly contradictory properties: on the one hand, its constituent layers exhibit highly similar dominant first principal components (PCs), but on the other hand, their representations, and their common first PC, are highly dissimilar across different random seeds. Our work seeks to reconcile these discrepant properties by investigating the origin of the block structure in relation to the data and training methods. By analyzing properties of the dominant PCs, we find that the block structure arises from dominant datapoints - a small group of examples that share similar image statistics (e.g. background color). However, the set of dominant datapoints, and the precise shared image statistic, can vary across random seeds. Thus, the block structure reflects meaningful dataset statistics, but is simultaneously unique to each model. Through studying hidden layer activations and creating synthetic datapoints, we demonstrate that these simple image statistics dominate the representational geometry of the layers inside the block structure. We explore how the phenomenon evolves through training, finding that the block structure takes shape early in training, but the underlying representations and the corresponding dominant datapoints continue to change substantially. Finally, we study the interplay between the block structure and different training mechanisms, introducing a targeted intervention to eliminate the block structure, as well as examining the effects of pretraining and Shake-Shake regularization.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 37 of its 47 references that are in the map. First posted on arXiv 15 Feb 2022.

Builds on

37 of 47 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  4. 2014Going Deeper with Convolutionscited by 4,241
  5. 2016Wide Residual Networkscited by 3,070
  6. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  7. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  8. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  9. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  10. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  11. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  12. 2019Benign Overfitting in Linear Regressioncited by 549
25 more
  1. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  2. 2018Rotation Equivariant CNNs for Digital Pathologycited by 422
  3. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  4. 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 368
  5. 2020What is being transferred in transfer learning?cited by 368
  6. 2018Insights on representational similarity in neural networks with canonical correlationcited by 324
  7. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 245
  8. 2017High-dimensional dynamics of generalization error in neural networkscited by 241
  9. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
  10. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  11. 2018To understand deep learning we need to understand kernel learningcited by 187
  12. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 183
  13. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
  14. 2020What shapes feature representations? Exploring datasets, architectures, and trainingcited by 126
  15. 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 111
  16. 2018A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillationcited by 110
  17. 2016Deep Learning for Identifying Metastatic Breast Cancercited by 93
  18. 2017Shake-Shake regularizationcited by 87
  19. 2017Measuring the tendency of CNNs to Learn Surface Statistical Regularitiescited by 87
  20. 2018Semantic Adversarial Examplescited by 86
  21. 2019Universality and individuality in neural dynamics across large populations of recurrent networkscited by 61
  22. 2017Detecting Cancer Metastases on Gigapixel Pathology Imagescited by 52
  23. 2013Some Improvements on Deep Convolutional Neural Network Based Image Classificationcited by 34
  24. 2019Probing the State of the Art: A Critical Look at Visual Representation Evaluationcited by 6
  25. 2019The effect of task and training on intermediate representations in convolutional neural networks revealed with modified RV similarity analysiscited by 5

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 2; 2024: 3; 2025: 3.

10 papers.

  1. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 131
  2. 2022Reliability of CKA as a Similarity Measure in Deep Learningcited by 87
  3. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  4. 2025Block-Recurrent Dynamics in Vision Transformerscited by 20
  5. 2022DORA: Exploring Outlier Representations in Deep Neural Networkscited by 14
  6. 2025Layers at Similar Depths Generate Similar Activations Across LLM Architecturescited by 12
  7. 2024Effective Layer Pruning Through Similarity Metric Perspectivecited by 6
  8. 2024Identifying Sub-networks in Neural Networks via Functionally Similar Representationscited by 0
  9. 2024Dimensionality-induced information loss of outliers in deep neural networkscited by 0
  10. 2025Revisiting the Relation Between Robustness and Universalitycited by 0