# On the Versatile Uses of Partial Distance Correlation in Deep Learning

arXiv: https://arxiv.org/abs/2207.09684 · HTML: https://mlsanity.com/paper/2207.09684/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 75 references that are in the map.
- First posted on arXiv 20 Jul 2022.

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)
- [What is being transferred in transfer learning?](https://mlsanity.com/paper/2008.11687/) (2020)
- [Insights on representational similarity in neural networks with canonical correlation](https://mlsanity.com/paper/1806.05759/) (2018)
- [Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth](https://mlsanity.com/paper/2010.15327/) (2020)
- [Emergence of Invariance and Disentanglement in Deep Representations](https://mlsanity.com/paper/1706.01350/) (2017)
- [Isolating Sources of Disentanglement in Variational Autoencoders](https://mlsanity.com/paper/1802.04942/) (2018)
- [Improving Black-box Adversarial Attacks with a Transfer-based Prior](https://mlsanity.com/paper/1906.06919/) (2019)
- [Fader Networks: Manipulating Images by Sliding Attributes](https://mlsanity.com/paper/1706.00409/) (2017)
- [Invariant Representations without Adversarial Training](https://mlsanity.com/paper/1805.09458/) (2018)
- [Repulsive Deep Ensembles are Bayesian](https://mlsanity.com/paper/2106.11642/) (2021)
- [Learning Controllable Fair Representations](https://mlsanity.com/paper/1812.04218/) (2018)
- [Weakly Supervised Disentanglement with Guarantees](https://mlsanity.com/paper/1910.09772/) (2019)
- [Differentiable Quality Diversity](https://mlsanity.com/paper/2106.03894/) (2021)

## Built on by (most cited first, 19 of 19)

- [Out-of-distribution Detection Learning with Unreliable Out-of-distribution Sources](https://mlsanity.com/paper/2311.03236/) (2023)
- [Neural Lineage](https://mlsanity.com/paper/2406.11129/) (2024)
- [Disentangling representations of retinal images with generative models](https://mlsanity.com/paper/2402.19186/) (2024)
- [Equiangular Basis Vectors](https://mlsanity.com/paper/2303.11637/) (2023)
- [Continual Learning From a Stream of APIs](https://mlsanity.com/paper/2309.00023/) (2023)
- [Audio Embeddings as Teachers for Music Classification](https://mlsanity.com/paper/2306.17424/) (2023)
- [Intrinsic Dimension Correlation: uncovering nonlinear connections in multimodal representations](https://mlsanity.com/paper/2406.15812/) (2024)
- [A Distance Correlation-Based Approach to Characterize the Effectiveness of Recurrent Neural Networks for Time Series Forecasting](https://mlsanity.com/paper/2307.15830/) (2023)
- [Benchmarking Dependence Measures to Prevent Shortcut Learning in Medical Imaging](https://mlsanity.com/paper/2407.18792/) (2024)
- [Structured Transformations for Stable and Interpretable Neural Computation](https://mlsanity.com/paper/2508.00127/) (2025)
- [Fair Domain Generalization: An Information-Theoretic View](https://mlsanity.com/paper/2507.05823/) (2025)
- [Interpreting Bias in the Neural Networks: A Peek Into Representational Similarity](https://mlsanity.com/paper/2211.07774/) (2022)
- [Understanding the Benefits of Image Augmentations](https://mlsanity.com/paper/2306.06254/) (2023)
- [Fair Sufficient Representation Learning](https://mlsanity.com/paper/2504.01030/) (2025)
- [Model Recycling Framework for Multi-Source Data-Free Supervised Transfer Learning](https://mlsanity.com/paper/2508.02039/) (2025)
- [Enhancing Knowledge Transfer in Hyperspectral Image Classification via Cross-scene Knowledge Integration](https://mlsanity.com/paper/2512.08989/) (2025)
- [Agreement Disagreement Guided Knowledge Transfer for Cross-Scene Hyperspectral Imaging](https://mlsanity.com/paper/2512.08990/) (2025)
- [Mitigating Shortcut Learning via Feature Disentanglement in Medical Imaging: A Benchmark Study](https://mlsanity.com/paper/2602.18502/) (2026)
- [From Layers to Networks: Comparing Neural Representations via Diffusion Geometry](https://mlsanity.com/paper/2605.15901/) (2026)

## Abstract (arXiv)

Comparing the functional behavior of neural network models, whether it is a single network over time or two (or more networks) during or post-training, is an essential step in understanding what they are learning (and what they are not), and for identifying strategies for regularization or efficiency improvements. Despite recent progress, e.g., comparing vision transformers to CNNs, systematic comparison of function, especially across different networks, remains difficult and is often carried out layer by layer. Approaches such as canonical correlation analysis (CCA) are applicable in principle, but have been sparingly used so far. In this paper, we revisit a (less widely known) from statistics, called distance correlation (and its partial variant), designed to evaluate correlation between feature spaces of different dimensions. We describe the steps necessary to carry out its deployment for large scale models -- this opens the door to a surprising array of applications ranging from conditioning one deep model w.r.t. another, learning disentangled representations as well as optimizing diverse models that would directly be more robust to adversarial attacks. Our experiments suggest a versatile regularizer (or constraint) with many advantages, which avoids some of the common difficulties one faces in such analyses. Code is at https://github.com/zhenxingjian/Partial_Distance_Correlation.