mlsanityData to 7 Oct 2026

Understanding the Inner Workings of Language Models Through Representation Dissimilarity

arXiv 2310.14993 ↗First posted 23 Oct 2023cs.LG cs.AI cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

As language models are applied to an increasing number of real-world applications, understanding their inner workings has become an important issue in model trust, interpretability, and transparency. In this work we show that representation dissimilarity measures, which are functions that measure the extent to which two model's internal representations differ, can be a valuable tool for gaining insight into the mechanics of language models. Among our insights are: (i) an apparent asymmetry in the internal representations of model using SoLU and GeLU activation functions, (ii) evidence that dissimilarity measures can identify and locate generalization properties of models that are invisible via in-distribution test set performance, and (iii) new evaluations of how language model features vary as width and depth are increased. Our results suggest that dissimilarity measures are a promising set of tools for shedding light on the inner workings of language models.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 24 of its 33 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

24 of 33 references.

References · 24

  1. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  4. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  5. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  6. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  7. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  8. 2019Designing and Interpreting Probes with Control Taskscited by 608
  9. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  10. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  11. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  12. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
12 more
  1. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 241
  2. 2014Understanding image representations by measuring their equivariance and equivalencecited by 240
  3. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  4. 2021Revisiting Model Stitching to Compare Neural Representationscited by 203
  5. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 165
  6. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  7. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  8. 2021Generalized Shape Metrics on Neural Representationscited by 109
  9. 2019BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performancecited by 89
  10. 2021Representation Topology Divergence: A Method for Comparing Neural Network Representationscited by 66
  11. 2022On the Symmetries of Deep Learning Models and their Internal Representationscited by 63
  12. 2023All Roads Lead to Rome? Exploring the Invariance of Transformers' Representationscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 6; 2026: 1.

12 papers.

  1. 2025Tracing the Representation Geometry of Language Models from Pretraining to Post-trainingcited by 33
  2. 2024Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval Augmented Generation Systemscited by 12
  3. 2025Layers at Similar Depths Generate Similar Activations Across LLM Architecturescited by 11
  4. 2025Representation Potentials of Foundation Models for Multimodal Alignment: A Surveycited by 10
  5. 2024Tending Towards Stability: Convergence Challenges in Small Language Modelscited by 7
  6. 2023Towards Measuring Representational Similarity of Large Language Modelscited by 5
  7. 2024Investigating Representation Universality: Case Study on Genealogical Representationscited by 4
  8. 2025Spectral Insights into Data-Oblivious Critical Layers in Large Language Modelscited by 4
  9. 2026Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibilitycited by 2
  10. 2024Peeling Back the Layers: An In-Depth Evaluation of Encoder Architectures in Neural News Recommenderscited by 1
  11. 2025Leveraging Intermediate Representations of Time Series Foundation Models for Anomaly Detectioncited by 1
  12. 2025Revisiting the Relation Between Robustness and Universalitycited by 0