mlsanityData to 14 Oct 2026

SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability

arXiv 1706.05806 ↗First posted 19 Jun 2017stat.ML cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

We propose a new technique, Singular Vector Canonical Correlation Analysis (SVCCA), a tool for quickly comparing two representations in a way that is both invariant to affine transform (allowing comparison between different layers and networks) and fast to compute (allowing more comparisons to be calculated than with previous methods). We deploy this tool to measure the intrinsic dimensionality of layers, showing in some cases needless over-parameterization; to probe learning dynamics throughout training, finding that networks converge to final representations from the bottom up; to show where class-specific information in networks is formed; and to suggest new training regimes that simultaneously save computation and overfit less. Code: https://github.com/google/svcca/

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Jun 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 3; 2023: 0; 2024: 2; 2025: 2; 2026: 5.

16 papers.

Sort:

By area:

  1. 2024On the Origins of Linear Representations in Large Language Modelscited by 79
  2. 2021AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuningcited by 48
  3. 2024Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Modelscited by 29
  4. 2021PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformerscited by 23
  5. 2025Knowing Before Saying: LLM Representations Encode Information About Chain-of-Thought Success Before Completioncited by 23
  6. 2022Egeria: Efficient DNN Training with Knowledge-Guided Layer Freezingcited by 22
  7. 2022DORA: Exploring Outlier Representations in Deep Neural Networkscited by 14
  8. 2021Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddingscited by 12
  9. 2026Ventriloquist LLMs: Linear Alignment of Late-Stage Representationscited by 4
  10. 2021Doing More by Doing Less: How Structured Partial Backpropagation Improves Deep Learning Clusterscited by 2
  11. 2025Intervening to Learn and Compose Causally Disentangled Representationscited by 1
  12. 2026Supervised Learning Has a Geometric Blind Spotcited by 1
4 more
  1. 2026Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspectivecited by 1
  2. 2022Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networkscited by 0
  3. 2026Hidden Activations are not Enough I: Knowledge Matrices as Higher Representationscited by 0
  4. 2026Learning Cross-Model Activation Alignments with Explicit Many-to-Many Layer Mapscited by 0