mlsanityData to 7 Oct 2026

On the Joint Interaction of Models, Data, and Features

arXiv 2306.04793 ↗First posted 7 Jun 2023cs.LG stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

Learning features from data is one of the defining characteristics of deep learning, but our theoretical understanding of the role features play in deep learning is still rudimentary. To address this gap, we introduce a new tool, the interaction tensor, for empirically analyzing the interaction between data and model through features. With the interaction tensor, we make several key observations about how features are distributed in data and how models with different random seeds learn different features. Based on these observations, we propose a conceptual framework for feature learning. Under this framework, the expected accuracy for a single hypothesis and agreement for a pair of hypotheses can both be derived in closed-form. We demonstrate that the proposed framework can explain empirically observed phenomena, including the recently discovered Generalization Disagreement Equality (GDE) that allows for estimating the generalization error with only unlabeled data. Further, our theory also provides explicit construction of natural data distributions that break the GDE. Thus, we believe this work provides valuable new insight into our understanding of feature learning.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 42 of its 65 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

42 of 65 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2016Densely Connected Convolutional Networkscited by 6,691
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2017Squeeze-and-Excitation Networkscited by 4,763
  9. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  10. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  11. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  12. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
30 more
  1. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  2. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  3. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  4. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  5. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  6. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  7. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  8. 2022Toy Models of Superpositioncited by 928
  9. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  10. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  11. 2020GANSpace: Discovering Interpretable GAN Controlscited by 612
  12. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  13. 2017Progressive Neural Architecture Searchcited by 483
  14. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  15. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  16. 2018Insights on representational similarity in neural networks with canonical correlationcited by 318
  17. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  18. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  19. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 215
  20. 2016Synthesizing the preferred inputs for neurons in neural networks via deep generator networkscited by 181
  21. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  22. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  23. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  24. 2021Assessing Generalization of SGD via Disagreementcited by 116
  25. 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
  26. 2022Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shiftcited by 94
  27. 2020Characterizing Structural Regularities of Labeled Data in Overparameterized Modelscited by 93
  28. 2021RegNet: Self-Regulated Network for Image Classificationcited by 44
  29. 2020Distributional Generalization: A New Kind of Generalizationcited by 41
  30. 2022A Note on "Assessing Generalization of SGD via Disagreement"cited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 0; 2026: 1.

5 papers.

  1. 2024InternalInspector I2: Robust Confidence Estimation in LLMs through Internal Statescited by 29
  2. 2024A Survey on Evaluation of Out-of-Distribution Generalizationcited by 20
  3. 2024DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticitycited by 7
  4. 2024OOD-Chameleon: Is Algorithm Selection for OOD Generalization Learnable?cited by 2
  5. 2026What Do Students Learn? A Feature-Level Analysis of Dark Knowledgecited by 0