mlsanityData to 14 Oct 2026

Contrastive-Difference CKA Reveals Concept-Specific Structural Alignment Across Language Model Architectures

arXiv 2606.16897 ↗First posted 15 Jun 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Do different LLM architectures encode high-level concepts in structurally compatible ways? We systematically characterize a geometric-functional universality dissociation: across multiple concept domains and architectural families, moderate geometric convergence coexists with near-perfect functional transfer. Using contrastive-difference CKA (CKA_Delta), a training-free diagnostic that computes kernel alignment on per-sample contrastive differences, we isolate concept-specific convergence from generic similarity -- achieving significant discrimination where standard CKA cannot. The dissociation replicates across all six concept domains we test (five with p <= 0.017 geometric discrimination and safety as a converging-functional trend, p = 0.08), including two non-instruction concepts (code-vs-NL, reasoning-vs-recall) validated without system prompts; a single 70B--70B pair provides an observational note that universality may strengthen with scale, requiring replication with additional >=70B models. We position CKA_Delta as a practical regime classifier and architectural outlier detector (Gemma: d = 1.08, AUC = 0.79) rather than an absolute transfer-accuracy predictor, providing a training-free diagnostic for cross-architecture concept monitoring.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 28 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

23 of 28 references.

References · 23

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2023Mistral 7Bcited by 3,164
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  5. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  6. 2022Toy Models of Superpositioncited by 971
  7. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  8. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  9. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  10. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  11. 2024The Platonic Representation Hypothesiscited by 382
  12. 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 345
11 more
  1. 2023Linear Representations of Sentiment in Large Language Modelscited by 163
  2. 2021Generalized Shape Metrics on Neural Representationscited by 112
  3. 2025Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignmentcited by 62
  4. 2024Towards Universality: Studying Mechanistic Similarity Across Language Model Architecturescited by 31
  5. 2025Transferring Linear Features Across Language Models With Model Stitchingcited by 21
  6. 2025Activation Space Interventions Can Be Transferred Between Large Language Modelscited by 19
  7. 2025Cross-model Transferability among Large Language Models on the Platonic Representations of Conceptscited by 12
  8. 2025Probing then Editing Response Personality of Large Language Modelscited by 9
  9. 2026On the Non-Identifiability of Steering Vectors in Large Language Modelscited by 8
  10. 2026Universal Refusal Circuits Across LLMs: Cross-Model Transfer via Trajectory Replay and Concept-Basis Reconstructioncited by 2
  11. 2026Controllable and explainable personality sliders for LLMs at inference timecited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Linear Concepts: Discovering and Aligning Non-Linear Concept Manifolds in Large Language Models
  2. 2026A Unifying Framework for Concept-Based Representational Similarity
  3. 2025Polarity-Aware Probing for Quantifying Latent Alignment in Language Models
  4. 2026Mechanistic Analysis of Alignment Algorithms in Language Models
  5. 2025LLM Probing with Contrastive Eigenproblems: Improving Understanding and Applicability of CCS
  6. 2026Beyond Language: Format-Agnostic Reasoning Subspaces in Large Language Models
  7. 2024Revisiting the Superficial Alignment Hypothesis
  8. 2025Semantic Convergence: Investigating Shared Representations Across Scaled LLMs