# Knowledge Consistency between Neural Networks and Beyond

arXiv: https://arxiv.org/abs/1908.01581 · HTML: https://mlsanity.com/paper/1908.01581/

## Key facts

- Cited by 16 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 5 Aug 2019.

## Built on by (most cited first, 16 of 16)

- [PMET: Precise Model Editing in a Transformer](https://mlsanity.com/paper/2308.08742/) (2023)
- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective](https://mlsanity.com/paper/2102.00650/) (2021)
- [We're Different, We're the Same: Creative Homogeneity Across LLMs](https://mlsanity.com/paper/2501.19361/) (2025)
- [Grounding Representation Similarity with Statistical Testing](https://mlsanity.com/paper/2108.01661/) (2021)
- [Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again](https://mlsanity.com/paper/2210.04427/) (2022)
- [GULP: a prediction-based metric between representations](https://mlsanity.com/paper/2210.06545/) (2022)
- [FAIRER: Fairness as Decision Rationale Alignment](https://mlsanity.com/paper/2306.15299/) (2023)
- [Instance-wise or Class-wise? A Tale of Neighbor Shapley for Concept-based Explanation](https://mlsanity.com/paper/2109.01369/) (2021)
- [Visualizing the Emergence of Intermediate Visual Patterns in DNNs](https://mlsanity.com/paper/2111.03505/) (2021)
- [Layerwise Change of Knowledge in Neural Networks](https://mlsanity.com/paper/2409.08712/) (2024)
- [ImageNet Pre-training also Transfers Non-Robustness](https://mlsanity.com/paper/2106.10989/) (2021)
- [A Closer Look at Branch Classifiers of Multi-exit Architectures](https://mlsanity.com/paper/2204.13347/) (2022)
- [Towards Theoretical Analysis of Transformation Complexity of ReLU DNNs](https://mlsanity.com/paper/2205.01940/) (2022)
- [Attesting Model Lineage by Consisted Knowledge Evolution with Fine-Tuning Trajectory](https://mlsanity.com/paper/2601.11683/) (2026)
- [Spectral Roll-off Points Variations: Exploring Useful Information in Feature Maps by Its Variations](https://mlsanity.com/paper/2102.00369/) (2021)

## Abstract (arXiv)

This paper aims to analyze knowledge consistency between pre-trained deep neural networks. We propose a generic definition for knowledge consistency between neural networks at different fuzziness levels. A task-agnostic method is designed to disentangle feature components, which represent the consistent knowledge, from raw intermediate-layer features of each neural network. As a generic tool, our method can be broadly used for different applications. In preliminary experiments, we have used knowledge consistency as a tool to diagnose representations of neural networks. Knowledge consistency provides new insights to explain the success of existing deep-learning techniques, such as knowledge distillation and network compression. More crucially, knowledge consistency can also be used to refine pre-trained networks and boost performance.