# Cross-Model Semantics in Representation Learning

arXiv: https://arxiv.org/abs/2508.03649 · HTML: https://mlsanity.com/paper/2508.03649/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 33 references that are in the map.
- First posted on arXiv 5 Aug 2025.

## Builds on (most cited first, 29 of 29)

- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [Relative representations enable zero-shot latent space communication](https://mlsanity.com/paper/2209.15430/) (2022)
- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding](https://mlsanity.com/paper/2303.04245/) (2023)
- [Human alignment of neural network representations](https://mlsanity.com/paper/2211.01201/) (2022)
- [On the Symmetries of Deep Learning Models and their Internal Representations](https://mlsanity.com/paper/2205.14258/) (2022)
- [Text-To-Concept (and Back) via Cross-Model Alignment](https://mlsanity.com/paper/2305.06386/) (2023)
- [Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment](https://mlsanity.com/paper/2502.03714/) (2025)
- [Latent Space Translation via Semantic Alignment](https://mlsanity.com/paper/2311.00664/) (2023)
- [Rosetta Neurons: Mining the Common Units in a Model Zoo](https://mlsanity.com/paper/2306.09346/) (2023)
- [The Neural Race Reduction: Dynamics of Abstraction in Gated Networks](https://mlsanity.com/paper/2207.10430/) (2022)
- [Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similarity](https://mlsanity.com/paper/2406.14479/) (2024)
- [Equivariant Deep Weight Space Alignment](https://mlsanity.com/paper/2310.13397/) (2023)
- [Understanding the Emergence of Multimodal Representation Alignment](https://mlsanity.com/paper/2502.16282/) (2025)
- [From Bricks to Bridges: Product of Invariances to Enhance Latent Space Communication](https://mlsanity.com/paper/2310.01211/) (2023)
- [Optimizers Qualitatively Alter Solutions And We Should Leverage This](https://mlsanity.com/paper/2507.12224/) (2025)
- [When Representations Align: Universality in Representation Learning Dynamics](https://mlsanity.com/paper/2402.09142/) (2024)
- [Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrum](https://mlsanity.com/paper/2307.14531/) (2023)
- [Expressive Monotonic Neural Networks](https://mlsanity.com/paper/2307.07512/) (2023)
- [Towards Exact Computation of Inductive Bias](https://mlsanity.com/paper/2406.15941/) (2024)
- [Towards a Learning Theory of Representation Alignment](https://mlsanity.com/paper/2502.14047/) (2025)
- [Teasing Apart Architecture and Initial Weights as Sources of Inductive Bias in Neural Networks](https://mlsanity.com/paper/2502.20237/) (2025)
- [Connecting Neural Models Latent Geometries with Relative Geodesic Representations](https://mlsanity.com/paper/2506.01599/) (2025)
- [Model Alignment Search](https://mlsanity.com/paper/2501.06164/) (2025)
- [Multimodal Representation Alignment for Cross-modal Information Retrieval](https://mlsanity.com/paper/2506.08774/) (2025)
- [Structured Transformations for Stable and Interpretable Neural Computation](https://mlsanity.com/paper/2508.00127/) (2025)
- [Latent Functional Maps: a spectral framework for representation alignment](https://mlsanity.com/paper/2406.14183/) (2024)
- [Understanding Learning Dynamics Through Structured Representations](https://mlsanity.com/paper/2508.02126/) (2025)

## Built on by (most cited first, 2 of 2)

- [Do Reasoning Models Enhance Embedding Models?](https://mlsanity.com/paper/2601.21192/) (2026)
- [A Stitch in Time Saves Nine: Preserving Policy Compatibility Under Perception Updates in End-to-End Autonomous Driving](https://mlsanity.com/paper/2606.21509/) (2026)

## Abstract (arXiv)

The internal representations learned by deep networks are often sensitive to architecture-specific choices, raising questions about the stability, alignment, and transferability of learned structure across models. In this paper, we investigate how structural constraints--such as linear shaping operators and corrective paths--affect the compatibility of internal representations across different architectures. Building on the insights from prior studies on structured transformations and convergence, we develop a framework for measuring and analyzing representational alignment across networks with distinct but related architectural priors. Through a combination of theoretical insights, empirical probes, and controlled transfer experiments, we demonstrate that structural regularities induce representational geometry that is more stable under architectural variation. This suggests that certain forms of inductive bias not only support generalization within a model, but also improve the interoperability of learned features across models. We conclude with a discussion on the implications of representational transferability for model distillation, modular learning, and the principled design of robust learning systems.