mlsanityData to 7 Oct 2026

Discovering and Explaining the Representation Bottleneck of DNNs

arXiv 2111.06236 ↗First posted 11 Nov 2021cs.LG cs.AI cs.CVcited by 42

Not on the map. Why?

Abstract (arXiv)

This paper explores the bottleneck of feature representations of deep neural networks (DNNs), from the perspective of the complexity of interactions between input variables encoded in DNNs. To this end, we focus on the multi-order interaction between input variables, where the order represents the complexity of interactions. We discover that a DNN is more likely to encode both too simple interactions and too complex interactions, but usually fails to learn interactions of intermediate complexity. Such a phenomenon is widely shared by different DNNs for different tasks. This phenomenon indicates a cognition gap between DNNs and human beings, and we call it a representation bottleneck. We theoretically prove the underlying reason for the representation bottleneck. Furthermore, we propose a loss to encourage/penalize the learning of interactions of specific complexities, and analyze the representation capacities of interactions of different complexities.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 31 of its 44 references that are in the map. First posted on arXiv 11 Nov 2021.

Builds on

31 of 44 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  7. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  8. 2017A Closer Look at Memorization in Deep Networkscited by 936
  9. 2017Exploring Generalization in Deep Learningcited by 595
  10. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  11. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  12. 2018AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networkscited by 319
19 more
  1. 2018Consistent Individualized Feature Attribution for Tree Ensemblescited by 279
  2. 2018xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systemscited by 277
  3. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  4. 2018Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approachcited by 181
  5. 2019The Shapley Taylor Interaction Indexcited by 143
  6. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 131
  7. 2019Explaining Deep Neural Networks with a Polynomial Time Algorithm for Shapley Values Approximationcited by 115
  8. 2020Explaining Explanations: Axiomatic Feature Interactions for Deep Networkscited by 103
  9. 2013On the number of response regions of deep feed forward networks with piece-wise linear activationscited by 94
  10. 2020The Hessian Penalty: A Weak Prior for Unsupervised Disentanglementcited by 90
  11. 2017Detecting Statistical Interactions from Neural Network Weightscited by 80
  12. 2020A Unified Approach to Interpreting and Boosting Adversarial Transferabilitycited by 71
  13. 2018Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principlecited by 70
  14. 2019Stiffness: A New Perspective on Generalization in Neural Networkscited by 46
  15. 2018Understanding training and generalization in deep learning by Fourier analysiscited by 43
  16. 2020Interpreting and Boosting Dropout from a Game-Theoretic Viewcited by 34
  17. 2020Feature Interaction Interpretability: A Case for Explaining Ad-Recommendation Systems via Neural Interaction Detectioncited by 32
  18. 2021A Game-Theoretic Taxonomy of Visual Concepts in DNNscited by 16
  19. 2021Interpreting Attributions and Interactions of Adversarial Attackscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 11; 2024: 13; 2025: 6; 2026: 4.

42 papers.

Sort:

By area:

  1. 2022MogaNet: Multi-order Gated Aggregation Networkcited by 59
  2. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  3. 2024shapiq: Shapley Interactions for Machine Learningcited by 42
  4. 2022Architecture-Agnostic Masked Image Modeling -- From ViT back to CNNcited by 31
  5. 2023Does a Neural Network Really Encode Symbolic Concepts?cited by 27
  6. 2023Demystifying Causal Features on Adversarial Examples and Causal Inoculation for Robust Network by Adversarial Instrumental Variable Regressioncited by 18
  7. 2023HarsanyiNet: Computing Accurate Shapley Values in a Single Forward Propagationcited by 16
  8. 2024Defining and Extracting generalizable interaction primitives from DNNscited by 14
  9. 2024Towards the Dynamics of a DNN Learning Symbolic Interactionscited by 13
  10. 2023Can the Inference Logic of Large Language Models be Disentangled into Symbolic Concepts?cited by 11
  11. 2022Defects of Convolutional Decoder Networks in Frequency Representationcited by 10
  12. 2023Technical Note: Defining and Quantifying AND-OR Interactions for Faithful and Concise Explanation of DNNscited by 10
30 more
  1. 2025INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Samplingcited by 10
  2. 2023Bayesian Neural Networks Avoid Encoding Complex and Perturbation-Sensitive Conceptscited by 9
  3. 2023Where We Have Arrived in Proving the Emergence of Sparse Symbolic Concepts in AI Modelscited by 8
  4. 2024Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic Viewcited by 7
  5. 2024Improving Semantic Control in Discrete Latent Spaces with Transformer Quantized Variational Autoencoderscited by 6
  6. 2024Identifying Important Group of Pixels using Interactionscited by 5
  7. 2022Discovering the Representation Bottleneck of Graph Neural Networkscited by 4
  8. 2024Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layers?cited by 4
  9. 2024Layerwise Change of Knowledge in Neural Networkscited by 4
  10. 2022Batch Normalization Is Blind to the First and Second Derivatives of the Losscited by 3
  11. 2023Understanding Data Augmentation from a Robustness Perspectivecited by 3
  12. 2022Game-Theoretic Understanding of Misclassificationcited by 1
  13. 2023Schema Inference for Interpretable Image Classificationcited by 1
  14. 2023Towards Attributions of Input Variables in a Coalitioncited by 1
  15. 2024Why pre-training is beneficial for downstream classification tasks?cited by 1
  16. 2024Evaluating the Correctness of Inference Patterns Used by LLMs for Judgmentcited by 1
  17. 2024EAPCR: A Universal Feature Extractor for Scientific Data without Explicit Feature Relation Patternscited by 1
  18. 2025Revisiting Generalization Power of a DNN in Terms of Symbolic Interactionscited by 1
  19. 2025Technical Report: Quantifying and Analyzing the Generalization Power of a DNNcited by 1
  20. 2025The Interaction Bottleneck of Deep Neural Networks: Discovery, Proof, and Modulationcited by 1
  21. 2026Cross-LLM Consistency in Inference: Evidence from Shared Interactionscited by 1
  22. 2022Explanation-based Counterfactual Retraining(XCR): A Calibration Method for Black-box Modelscited by 0
  23. 2022MixBoost: Improving the Robustness of Deep Neural Networks by Boosting Data Augmentationcited by 0
  24. 2023Structured Network Pruning by Measuring Filter-wise Interactionscited by 0
  25. 2024Disentangling Regional Primitives for Image Generationcited by 0
  26. 2025Conceptualizing Multi-scale Wavelet Attention and Ray-based Encoding for Human-Object Interaction Detectioncited by 0
  27. 2025SpaRTAN: Spatial Reinforcement Token-based Aggregation Network for Visual Recognitioncited by 0
  28. 2026Understanding Generalization through Decision Pattern Shiftcited by 0
  29. 2026Reconciling Contradictory Views on the Effectiveness of SFT in LLMs: An Interaction Perspectivecited by 0
  30. 2026Exactness at Inference: A Representational Criterion for Out-of-Distribution Generalizationcited by 0