mlsanityData to 14 Oct 2026

When Cosine Similarity Fails to Reflect Linearly Accessible Structure in Dialogue Models

arXiv 2609.22522 ↗First posted 18 Sep 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Cosine similarity is widely used to analyze transformer representations, implicitly assuming that similarity reflects task-relevant structure. We study when this assumption fails in dialogue-conditioned large language models. Across three 7-8B chat-tuned models, ambient cosine similarity substantially underestimates linearly decodable persona structure on the same hidden states; numerically, linear probe AUC is in the 0.73-0.97 range while cosine kNN is in the 0.56-0.77 range on a 30-class task. A low-dimensional supervised subspace recovers much of this gap, whereas a matched-rank PCA subspace does not and in some cases degrades performance. This mismatch is regime-dependent: it is absent in single-sentence sentiment classification (SST-5), and a matched-cardinality control rules out attribute cardinality as a confound. The gap does not systematically increase across dialogue turns, and the task-aligned subspace remains stable over time. However, two of three models violate a pre-registered within-subspace separability invariance criterion (|Delta AUC| <= 0.03), and one model violates a pre-registered turn-invariance criterion (|Delta L| <= 0.05). These results show that cosine similarity can fail to reflect task-aligned structure in dialogue representations even when that structure is linearly accessible.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 26 references that are in the map. First posted on arXiv 18 Sep 2026.

Builds on

24 of 26 references.

References · 24

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  3. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  4. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  5. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  6. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  7. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  8. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  9. 2019Designing and Interpreting Probes with Control Taskscited by 640
  10. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 455
  11. 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 345
  12. 2023LEACE: Perfect linear concept erasure in closed formcited by 267
12 more
  1. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 255
  2. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 212
  3. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 212
  4. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 156
  5. 2024Analyzing the Generalization and Reliability of Steering Vectorscited by 143
  6. 2026The Chameleon's Limit: Investigating Persona Collapse and Homogenization in Large Language Modelscited by 8
  7. 2026A Systematic Analysis of the Impact of Persona Steering on LLM Capabilitiescited by 5
  8. 2026Restoring Heterogeneity in LLM-based Social Simulation: An Audience Segmentation Approachcited by 4
  9. 2025The Geometry of Persona: Disentangling Personality from Reasoning in Large Language Modelscited by 3
  10. 2026Stable Behavior, Limited Variation: Persona Validity in LLM Agents for Urban Sentiment Perceptioncited by 2
  11. 2026The Pragmatic Persona: Discovering LLM Persona through Bridging Inferencecited by 1
  12. 2026Characterizing the Consistency of the Emergent Misalignment Personacited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Layer Relevance in Large Language Models Beyond Cosine Similarity
  2. 2026Aligning Backchannel and Dialogue Context Representations via Contrastive LLM Fine-Tuning
  3. 2025Towards Stable and Personalised Profiles for Lexical Alignment in Spoken Human-Agent Dialogue
  4. 2025Neural Correlates of Language Models Are Specific to Human Language
  5. 2025Semantic Convergence: Investigating Shared Representations Across Scaled LLMs
  6. 2026PersoDPO: Scalable Preference Optimization for Instruction-Adherent, Persona-Grounded Dialogue via Multi-LLM Evaluation
  7. 2026Cosine Misleads: Auxiliary Losses Reshape Vision Language Models, Not Their Latents
  8. 2025Cat, Rat, Meow: On the Alignment of Language Model and Human Term-Similarity Judgments