# SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability

arXiv: https://arxiv.org/abs/1706.05806 · HTML: https://mlsanity.com/paper/1706.05806/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jun 2017.

## Built on by (most cited first, 16 of 16)

- [On the Origins of Linear Representations in Large Language Models](https://mlsanity.com/paper/2403.03867/) (2024)
- [AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning](https://mlsanity.com/paper/2102.01386/) (2021)
- [Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Models](https://mlsanity.com/paper/2402.09236/) (2024)
- [PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers](https://mlsanity.com/paper/2102.03161/) (2021)
- [Knowing Before Saying: LLM Representations Encode Information About Chain-of-Thought Success Before Completion](https://mlsanity.com/paper/2505.24362/) (2025)
- [Egeria: Efficient DNN Training with Knowledge-Guided Layer Freezing](https://mlsanity.com/paper/2201.06227/) (2022)
- [DORA: Exploring Outlier Representations in Deep Neural Networks](https://mlsanity.com/paper/2206.04530/) (2022)
- [Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings](https://mlsanity.com/paper/2103.02886/) (2021)
- [Ventriloquist LLMs: Linear Alignment of Late-Stage Representations](https://mlsanity.com/paper/2603.18908/) (2026)
- [Doing More by Doing Less: How Structured Partial Backpropagation Improves Deep Learning Clusters](https://mlsanity.com/paper/2111.10672/) (2021)
- [Intervening to Learn and Compose Causally Disentangled Representations](https://mlsanity.com/paper/2507.04754/) (2025)
- [Supervised Learning Has a Geometric Blind Spot](https://mlsanity.com/paper/2604.21395/) (2026)
- [Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective](https://mlsanity.com/paper/2607.18026/) (2026)
- [Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networks](https://mlsanity.com/paper/2201.07934/) (2022)
- [Hidden Activations are not Enough I: Knowledge Matrices as Higher Representations](https://mlsanity.com/paper/2609.34166/) (2026)
- [Learning Cross-Model Activation Alignments with Explicit Many-to-Many Layer Maps](https://mlsanity.com/paper/2610.09058/) (2026)

## Abstract (arXiv)

We propose a new technique, Singular Vector Canonical Correlation Analysis (SVCCA), a tool for quickly comparing two representations in a way that is both invariant to affine transform (allowing comparison between different layers and networks) and fast to compute (allowing more comparisons to be calculated than with previous methods). We deploy this tool to measure the intrinsic dimensionality of layers, showing in some cases needless over-parameterization; to probe learning dynamics throughout training, finding that networks converge to final representations from the bottom up; to show where class-specific information in networks is formed; and to suggest new training regimes that simultaneously save computation and overfit less. Code: https://github.com/google/svcca/