mlsanityData to 14 Oct 2026

Representation Measurements Under Function-Preserving Reparameterizations

arXiv 2608.27020 ↗First posted 27 Aug 2026stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Hidden coordinates are not uniquely determined by a language model's input--output function, so representation-derived measurements should be invariant to function-preserving changes of basis. This study shows that column-permutation parallel analysis violates function-preserving reparameterization invariance because its reference distribution and selected component count can change while the model function and observed covariance spectrum remain fixed. More generally, a data-internal reference procedure cannot simultaneously preserve every coordinate marginal, remain orthogonally equivariant, and remove cross-coordinate covariance. Empirically, across five models, three retrieval domains, and 75 transformations, median component-count disagreement is 0.79 and median fixed-threshold decision disagreement is 0.26. A centering-only control isolates the reference-driven effect, with 1,141 of 1,200 component counts changing despite an unchanged observed spectrum, whereas independent parallel analysis seeds change none of the corresponding decisions. By contrast, orthogonally invariant comparator scores remain numerically stable with similar held-out discrimination. Together, these results show that parallel analysis-derived component counts and decisions can reflect hidden-coordinate choice rather than a well-defined property of the model.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 30 references that are in the map. First posted on arXiv 27 Aug 2026.

Builds on

21 of 30 references.

References · 21

  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  2. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  3. 2019Designing and Interpreting Probes with Control Taskscited by 640
  4. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  5. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  6. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  7. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  8. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  9. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 265
  10. 2020Information-Theoretic Probing for Linguistic Structurecited by 207
  11. 2020On Linear Identifiability of Learned Representationscited by 105
  12. 2025Sparse Autoencoders Do Not Find Canonical Units of Analysiscited by 88
9 more
  1. 2024Sufficient Context: A New Lens on Retrieval Augmented Generation Systemscited by 72
  2. 2022On the Symmetries of Deep Learning Models and their Internal Representationscited by 64
  3. 2021On the Pitfalls of Analyzing Individual Neurons in Language Modelscited by 62
  4. 2024RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generationcited by 61
  5. 2024Emergence of a High-Dimensional Abstraction Phase in Language Transformerscited by 56
  6. 2024ReSi: A Comprehensive Benchmark for Representational Similarity Measurescited by 19
  7. 2024Geometric Signatures of Compositionality Across a Language Model's Lifetimecited by 16
  8. 2017Permutation methods for factor analysis and PCAcited by 9
  9. 2025When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspectivecited by 4

Built on by

Not cited yet.