mlsanityData to 7 Oct 2026

The Loss Kernel: A Geometric Probe for Deep Learning Interpretability

arXiv 2509.26537 ↗First posted 30 Sep 2025cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

We introduce the loss kernel, an interpretability method for measuring similarity between data points according to a trained neural network. The kernel is the covariance matrix of per-sample losses computed under a distribution of low-loss-preserving parameter perturbations. We first validate our method on a synthetic multitask problem, showing it separates inputs by task as predicted by theory. We then apply this kernel to Inception-v1 to visualize the structure of ImageNet, and we show that the kernel's structure aligns with the WordNet semantic hierarchy. This establishes the loss kernel as a practical tool for interpretability and data attribution.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 20 of its 46 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

20 of 46 references.

References · 20

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  3. 2014Going Deeper with Convolutionscited by 4,231
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  5. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  6. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  7. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  8. 2022Toy Models of Superpositioncited by 928
  9. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  10. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
  11. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 50
  12. 2017Covariances, Robustness, and Variational Bayescited by 39
8 more
  1. 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 39
  2. 2023Dynamical versus Bayesian Phase Transitions in a Toy Model of Superpositioncited by 28
  3. 2025You Are What You Eat -- AI Alignment Requires Understanding How Data Shapes Structure and Generalisationcited by 15
  4. 2025Dynamics of Transient Structure in In-Context Linear Regression Transformerscited by 12
  5. 2025Training Data Attribution (TDA): Examining Its Adoption & Use Casescited by 4
  6. 2025Structure Development in List-Sorting Transformerscited by 4
  7. 2023W-Kernel and Its Principal Space for Frequentist Evaluation of Bayesian Estimatorscited by 3
  8. 2025Programs as Singularitiescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2025Influence Dynamics and Stagewise Data Attributioncited by 5
  2. 2025Feature Identification via the Empirical NTKcited by 4
  3. 2026Towards Worst-Case Guarantees with Scale-Aware Interpretabilitycited by 3
  4. 2026From Density Matrices to Phase Transitions in Deep Learning: Spectral Early Warnings and Interpretabilitycited by 3
  5. 2026Mechanistic Anomaly Detection via Functional Attributioncited by 1
  6. 2026Susceptibilities and Patterning: A Primer on Linear Response in Bayesian Learningcited by 1
  7. 2026Differentiable Kernel Ridge Regression for Deep Learning Pipelinescited by 0
  8. 2026Patterning in Practice: Debiasing Reward Models with Susceptibilitiescited by 0