# Mean-Pooled Cosine Similarity is Not Length-Invariant: Theory and Cross-Domain Evidence for a Length-Invariant Alternative

arXiv: https://arxiv.org/abs/2605.07345 · HTML: https://mlsanity.com/paper/2605.07345/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 16 references that are in the map.
- First posted on arXiv 8 May 2026.

## Builds on (most cited first, 11 of 11)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [OctoPack: Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2308.07124/) (2023)
- [Reliability of CKA as a Similarity Measure in Deep Learning](https://mlsanity.com/paper/2210.16156/) (2022)

## Abstract (arXiv)

Mean-pooled cosine similarity is the default metric for comparing neural representations across languages, modalities, and tasks. We establish that this metric is not length-invariant: under the anisotropy that characterizes modern transformer representations, mean-pooled cosine grows monotonically in sequence length, independent of representational content. Empirically, on HumanEvalPack across four code LLMs, the length ratio alone explains R2 = 0.52--0.75 of cross-language "Python proximity," while AST depth and shared-token fraction add less than 3% of explained variance beyond length. Substituting Centered Kernel Alignment (CKA) reduces explained variance by 83% and reverses the sign of the length coefficient (βlen: +0.86 → -0.37). The same pattern holds in Mistral-7B on parallel WMT pairs (R2 = 0.23 EN-FR, R2 = 0.33 EN-DE for cosine; R2 < 0.01 for CKA). In CLIP ViT-B/32, mean-pooling reduces the length effect relative to EOS-pooling (R2: 0.21 → <0.01), as predicted by the theory's dependence on anisotropy. We argue that length-invariant metrics such as CKA should be the default for cross-representation comparisons, and that recent claims of cross-lingual representational convergence built on mean-pooled cosine warrant re-examination.