mlsanityData to 7 Oct 2026

Similarity Analysis of Contextual Word Representation Models

arXiv 2005.01172 ↗First posted 3 May 2020cs.CLcited by 54

Not on the map. Why?

Abstract (arXiv)

This paper investigates contextual word representation models from the lens of similarity analysis. Given a collection of trained models, we measure the similarity of their internal representations and attention. Critically, these models come from vastly different architectures. We use existing and novel similarity measures that aim to gauge the level of localization of information in the deep models, and facilitate the investigation of which design factors affect model similarity, without requiring any external linguistic annotation. The analysis reveals that models within the same family are more similar to one another, as may be expected. Surprisingly, different architectures have rather similar representations, but different individual neurons. We also observed differences in information localization in lower and higher layers and found that higher layers are more affected by fine-tuning on downstream tasks.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 5; 2023: 9; 2024: 12; 2025: 4; 2026: 6.

54 papers.

Sort:

By area:

  1. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  2. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 140
  3. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  4. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  5. 2021Similarity and Matching of Neural Network Representationscited by 77
  6. 2021Representation Topology Divergence: A Method for Comparing Neural Network Representationscited by 66
  7. 2022Discovering Latent Concepts Learned in BERTcited by 56
  8. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  9. 2024Do Vision and Language Encoders Represent the World Similarly?cited by 44
  10. 2024Who's asking? User personas and the mechanics of latent misalignmentcited by 40
  11. 2022Analyzing Encoded Concepts in Transformer Language Modelscited by 37
  12. 2021How transfer learning impacts linguistic knowledge in deep NLP models?cited by 32
38 more
  1. 2022ModelDiff: A Framework for Comparing Learning Algorithmscited by 30
  2. 2021Fine-Tuned Transformers Show Clusters of Similar Representations Across Layerscited by 29
  3. 2024Large Language Models Relearn Removed Conceptscited by 26
  4. 2024Mechanistic?cited by 25
  5. 2024Looking Right is Sometimes Right: Investigating the Capabilities of Decoder-only LLMs for Sequence Labelingcited by 24
  6. 2024Constructing Benchmarks and Interventions for Combating Hallucinations in LLMscited by 19
  7. 2021Of Non-Linearity and Commutativity in BERTcited by 16
  8. 2021On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformerscited by 14
  9. 2024Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval Augmented Generation Systemscited by 12
  10. 2025Layers at Similar Depths Generate Similar Activations Across LLM Architecturescited by 11
  11. 2022Representation Projection Invariance Mitigates Representation Collapsecited by 10
  12. 2024What Do Self-Supervised Speech and Speaker Models Learn? New Findings From a Cross Model Layer-Wise Analysiscited by 10
  13. 2021How Familiar Does That Sound? Cross-Lingual Representational Similarity Analysis of Acoustic Word Embeddingscited by 8
  14. 2023On Robustness of Finetuned Transformer-based NLP Modelscited by 8
  15. 2021Effect of Post-processing on Contextualized Word Representationscited by 7
  16. 2022On the Effect of Pre-training for Transformer in Different Modality on Offline Reinforcement Learningcited by 7
  17. 2024Tending Towards Stability: Convergence Challenges in Small Language Modelscited by 7
  18. 2025Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretrainingcited by 6
  19. 2021Morph Call: Probing Morphosyntactic Content of Multilingual Transformerscited by 5
  20. 2023CRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Modelcited by 5
  21. 2023Towards Measuring Representational Similarity of Large Language Modelscited by 5
  22. 2026Old Habits Die Hard: How Conversational History Geometrically Traps LLMscited by 5
  23. 2021RuSentEval: Linguistic Source, Encoder Force!cited by 4
  24. 2021Fine-grained Interpretation and Causation Analysis in Deep NLP Modelscited by 4
  25. 2023ContraSim -- Analyzing Neural Representations Based on Contrastive Learningcited by 4
  26. 2023N2G: A Scalable Approach for Quantifying Interpretable Neuron Representations in Large Language Modelscited by 4
  27. 2023What Happens During Finetuning of Vision Transformers: An Invariance Based Investigationcited by 3
  28. 2026Differentiable Faithfulness Alignment for Cross-Model Circuit Transfercited by 3
  29. 2021Neural Supervised Domain Adaptation by Augmenting Pre-trained Models with Random Unitscited by 2
  30. 2021On Neurons Invariant to Sentence Structural Changes in Neural Machine Translationcited by 2
  31. 2021Using Distributional Principles for the Semantic Study of Contextual Language Modelscited by 1
  32. 2024What Makes Two Language Models Think Alike?cited by 1
  33. 2025Origin Tracer: A Method for Detecting LoRA Fine-Tuning Origins in LLMscited by 1
  34. 2025A Representation Level Analysis of NMT Model Robustness to Grammatical Errorscited by 1
  35. 2026Do LLMs and VLMs Share Neurons for Inference? Evidence and Mechanisms of Cross-Modal Transfercited by 1
  36. 2026A Causal Language Modeling Detour Improves Encoder Continued Pretrainingcited by 1
  37. 2021On Isotropy Calibration of Transformerscited by 0
  38. 2023Manifold-Preserving Transformers are Effective for Short-Long Range Encodingcited by 0

50 most cited of 54 All 54 (JSON)