A Fiber Criterion for Representation Identifiability in Supervised Learning
Not on the map. Why?
Abstract (arXiv)
Supervised learning evaluates predictors through their input-output behavior. When a predictor is implemented as a composition f=ccirc h, supervised evidence constrains the composite map f but need not determine the representation-head factorization (h,c). This paper formalizes the resulting representation-level identifiability problem: for a class of admissible representation-head pairs, a representation property is identifiable from the induced predictor exactly when it is constant on the fibers of the projection (h,c)mapsto ccirc h, equivalently when it descends to a well-defined property of the predictor. Predictor-preserving augmentation gives a canonical obstruction: auxiliary information can be appended to a representation while the head ignores it, leaving the predictor unchanged but altering properties such as minimality, compression, invariance, equivariance, nuisance information, or semantic accessibility. This construction separates representation identifiability from optimization and finite-sample estimation. Finite-sample diagnostics illustrate, rather than prove, the criterion: exact algebraic witnesses hold the predictor fixed while changing representation diagnostics, and matched-performance Waterbirds models show that different constraints can select different representations at similar supervised performance. The results clarify that representation-level claims require assumptions, objectives, measurements, or inductive biases beyond supervised predictive behavior alone.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 37 of its 50 references that are in the map. First posted on arXiv 31 May 2026.
Builds on
37 of 50 references.
References · 37
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
- 2015Domain-Adversarial Training of Neural Networkscited by 3,023
- 2020Shortcut Learning in Deep Neural Networkscited by 1,928
- 2019Invariant Risk Minimizationcited by 1,832
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2016Understanding deep learning requires rethinking generalizationcited by 1,657
- 2016Understanding intermediate layers using linear classifier probescited by 1,279
- 2016Group Equivariant Convolutional Networkscited by 1,154
- 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
- 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
25 more
- 2016Deep Variational Information Bottleneckcited by 915
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
- 2015Deep Learning and the Information Bottleneck Principlecited by 793
- 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
- 2018Disentangling by Factorisingcited by 614
- 2019Designing and Interpreting Probes with Control Taskscited by 608
- 2017Exploring Generalization in Deep Learningcited by 595
- 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
- 2015Causal inference using invariant prediction: identification and confidence intervalscited by 533
- 2019Variational Autoencoders and Nonlinear ICA: A Unifying Frameworkcited by 494
- 2018Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learningcited by 278
- 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 213
- 2020Invariant Risk Minimization Gamescited by 188
- 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 131
- 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
- 2021Does Invariant Risk Minimization Capture Invariance?cited by 116
- 2020On Linear Identifiability of Learned Representationscited by 103
- 2021Invariant Information Bottleneck for Domain Generalizationcited by 86
- 2021Conditional probing: measuring usable information beyond a baselinecited by 61
- 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 33
- 2026Gauge-invariant representation holonomycited by 7
- 2020A general theory of identificationcited by 4
- 2026Statistical and structural identifiability in representation learningcited by 4
- 2026Training Memory in Deep Neural Networks: Mechanisms, Evidence, and Measurement Gapscited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.