# Towards the Dynamics of a DNN Learning Symbolic Interactions

arXiv: https://arxiv.org/abs/2407.19198 · HTML: https://mlsanity.com/paper/2407.19198/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 46 references that are in the map.
- First posted on arXiv 27 Jul 2024.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Understanding Neural Networks Through Deep Visualization](https://mlsanity.com/paper/1506.06579/) (2015)
- [Distilling a Neural Network Into a Soft Decision Tree](https://mlsanity.com/paper/1711.09784/) (2017)
- [The Shapley Taylor Interaction Index](https://mlsanity.com/paper/1902.05622/) (2019)
- [A Unified Approach to Interpreting and Boosting Adversarial Transferability](https://mlsanity.com/paper/2010.04055/) (2020)
- [Discovering and Explaining the Representation Bottleneck of DNNs](https://mlsanity.com/paper/2111.06236/) (2021)
- [Defining and Quantifying the Emergence of Sparse Concepts in DNNs](https://mlsanity.com/paper/2111.06206/) (2021)
- [Interpreting and Boosting Dropout from a Game-Theoretic View](https://mlsanity.com/paper/2009.11729/) (2020)
- [Explainable Neural Networks based on Additive Index Models](https://mlsanity.com/paper/1806.01933/) (2018)
- [Does a Neural Network Really Encode Symbolic Concepts?](https://mlsanity.com/paper/2302.13080/) (2023)
- [Explaining Generalization Power of a DNN Using Interactive Concepts](https://mlsanity.com/paper/2302.13091/) (2023)
- [A Game-Theoretic Taxonomy of Visual Concepts in DNNs](https://mlsanity.com/paper/2106.10938/) (2021)
- [Defining and Extracting generalizable interaction primitives from DNNs](https://mlsanity.com/paper/2401.16318/) (2024)
- [Proving Common Mechanisms Shared by Twelve Methods of Boosting Adversarial Transferability](https://mlsanity.com/paper/2207.11694/) (2022)
- [Technical Note: Defining and Quantifying AND-OR Interactions for Faithful and Concise Explanation of DNNs](https://mlsanity.com/paper/2304.13312/) (2023)
- [Considerations When Learning Additive Explanations for Black-Box Models](https://mlsanity.com/paper/1801.08640/) (2018)
- [A Hypothesis for the Aesthetic Appreciation in Neural Networks](https://mlsanity.com/paper/2108.02646/) (2021)
- [Explaining How a Neural Network Play the Go Game and Let People Learn](https://mlsanity.com/paper/2310.09838/) (2023)
- [Two-Phase Dynamics of Interactions Explains the Starting Point of a DNN Learning Over-Fitted Features](https://mlsanity.com/paper/2405.10262/) (2024)

## Built on by (most cited first, 13 of 13)

- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [Tracking the Feature Dynamics in LLM Training: A Mechanistic Study](https://mlsanity.com/paper/2412.17626/) (2024)
- [ProxySPEX: Inference-Efficient Interpretability via Sparse Feature Interactions in LLMs](https://mlsanity.com/paper/2505.17495/) (2025)
- [SPEX: Scaling Feature Interaction Explanations for LLMs](https://mlsanity.com/paper/2502.13870/) (2025)
- [Physics of Skill Learning](https://mlsanity.com/paper/2501.12391/) (2025)
- [Revisiting Generalization Power of a DNN in Terms of Symbolic Interactions](https://mlsanity.com/paper/2502.10162/) (2025)
- [Technical Report: Quantifying and Analyzing the Generalization Power of a DNN](https://mlsanity.com/paper/2505.06993/) (2025)
- [textitFocaLogic: Logic-Based Interpretation of Visual Model Decisions](https://mlsanity.com/paper/2601.12049/) (2026)
- [Cross-LLM Consistency in Inference: Evidence from Shared Interactions](https://mlsanity.com/paper/2606.08129/) (2026)
- [Disentangling Regional Primitives for Image Generation](https://mlsanity.com/paper/2410.04421/) (2024)
- [Reconciling Contradictory Views on the Effectiveness of SFT in LLMs: An Interaction Perspective](https://mlsanity.com/paper/2605.17967/) (2026)
- [A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions](https://mlsanity.com/paper/2607.08776/) (2026)
- [Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions](https://mlsanity.com/paper/2608.18539/) (2026)

## Abstract (arXiv)

This study proves the two-phase dynamics of a deep neural network (DNN) learning interactions. Despite the long disappointing view of the faithfulness of post-hoc explanation of a DNN, a series of theorems have been proven in recent years to show that for a given input sample, a small set of interactions between input variables can be considered as primitive inference patterns that faithfully represent a DNN's detailed inference logic on that sample. Particularly, Zhang et al. have observed that various DNNs all learn interactions of different complexities in two distinct phases, and this two-phase dynamics well explains how a DNN changes from under-fitting to over-fitting. Therefore, in this study, we mathematically prove the two-phase dynamics of interactions, providing a theoretical mechanism for how the generalization power of a DNN changes during the training process. Experiments show that our theory well predicts the real dynamics of interactions on different DNNs trained for various tasks.