mlsanityData to 7 Oct 2026

Towards Measuring Representational Similarity of Large Language Models

arXiv 2312.02730 ↗First posted 5 Dec 2023cs.LG cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Understanding the similarity of the numerous released large language models (LLMs) has many uses, e.g., simplifying model selection, detecting illegal model reuse, and advancing our understanding of what makes LLMs perform well. In this work, we measure the similarity of representations of a set of LLMs with 7B parameters. Our results suggest that some LLMs are substantially different from others. We identify challenges of using representational similarity measures that suggest the need of careful study of similarity scores to avoid false conclusions.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 33 of its 48 references that are in the map. First posted on arXiv 5 Dec 2023.

Builds on

33 of 48 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2018Deep contextualized word representationscited by 2,285
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
21 more
  1. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  2. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  3. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  4. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  5. 2022Galactica: A Large Language Model for Sciencecited by 736
  6. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  7. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  8. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  9. 2023Studying Large Language Model Generalization with Influence Functionscited by 311
  10. 2016Diachronic Word Embeddings Reveal Statistical Laws of Semantic Changecited by 219
  11. 2022Relative representations enable zero-shot latent space communicationcited by 175
  12. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  13. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  14. 2019BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performancecited by 89
  15. 2020Similarity Analysis of Contextual Word Representation Modelscited by 54
  16. 2018On the Dimensionality of Word Embeddingcited by 46
  17. 2019Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brainscited by 37
  18. 2022ModelDiff: A Framework for Comparing Learning Algorithmscited by 30
  19. 2020The Effects of Randomness on the Stability of Node Embeddingscited by 14
  20. 2023Understanding the Inner Workings of Language Models Through Representation Dissimilaritycited by 12
  21. 2022Understanding Weight Similarity of Neural Networks via Chain Normalization Rule and Hypothesis-Training-Testingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2024Objective drives the consistency of representational similarity across datasetscited by 16
  2. 2025Layers at Similar Depths Generate Similar Activations Across LLM Architecturescited by 11
  3. 2025Linear Representation Transferability Hypothesis: Leveraging Small Models to Steer Large Modelscited by 6
  4. 2025Causes and Consequences of Representational Similarity in Machine Learning Modelscited by 1
  5. 2026Do Reasoning Models Enhance Embedding Models?cited by 0