Order-Level Attention Similarity Across Language Models: A Latent Commonality
Not on the map. Why?
Abstract (arXiv)
In this paper, we explore an important yet previously neglected question: Do context aggregation patterns across Language Models (LMs) share commonalities? While some works have investigated context aggregation or attention weights in LMs, they typically focus on individual models or attention heads, lacking a systematic analysis across multiple LMs to explore their commonalities. In contrast, we focus on the commonalities among LMs, which can deepen our understanding of LMs and even facilitate cross-model knowledge transfer. In this work, we introduce the Order-Level Attention (OLA) derived from the order-wise decomposition of Attention Rollout and reveal that the OLA at the same order across LMs exhibits significant similarities. Furthermore, we discover an implicit mapping between OLA and syntactic knowledge. Based on these two findings, we propose the Transferable OLA Adapter (TOA), a training-free cross-LM adapter transfer method. Specifically, we treat the OLA as a unified syntactic feature representation and train an adapter that takes OLA as input. Due to the similarities in OLA across LMs, the adapter generalizes to unseen LMs without requiring any parameter updates. Extensive experiments demonstrate that TOA's cross-LM generalization effectively enhances the performance of unseen LMs. Code is available at https://github.com/jinglin-liang/OLAS.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 49 of its 61 references that are in the map. First posted on arXiv 7 Nov 2025.
Builds on
49 of 61 references.
References · 49
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2023GPT-4 Technical Reportcited by 18,132
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2019Root Mean Square Layer Normalizationcited by 1,425
- 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
- 2019Attention is not Explanationcited by 916
- 2020Quantifying Attention Flow in Transformerscited by 759
- 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
- 2019Attention is not not Explanationcited by 629
37 more
- 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
- 2019Is Attention Interpretable?cited by 387
- 2019Axial Attention in Multidimensional Transformerscited by 350
- 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 269
- 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
- 2022Relative representations enable zero-shot latent space communicationcited by 175
- 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
- 2022Knowledge Distillation with the Reused Teacher Classifiercited by 140
- 2020Higher-Order Explanations of Graph Neural Networks via Relevant Walkscited by 138
- 2019On Identifiability in Transformerscited by 115
- 2020Explaining Explanations: Axiomatic Feature Interactions for Deep Networkscited by 103
- 2018Context-Aware Neural Machine Translation Learns Anaphora Resolutioncited by 97
- 2017Detecting Statistical Interactions from Neural Network Weightscited by 80
- 2020Understanding SSIMcited by 75
- 2022Measuring the Mixing of Contextual Information in the Transformercited by 63
- 2021Incorporating Residual and Normalization Layers into Analysis of Masked Language Modelscited by 50
- 2020Attention Flows: Analyzing and Comparing Attention Mechanisms in Language Modelscited by 49
- 2020Towards Transparent and Explainable Attention Modelscited by 47
- 2022GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformerscited by 44
- 2023Disentangling Writer and Character Styles for Handwriting Generationcited by 35
- 2020Building and Interpreting Deep Similarity Modelscited by 34
- 2024AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Mergingcited by 34
- 2023Analyzing Feed-Forward Blocks in Transformers through the Lens of Attention Mapscited by 33
- 2023DecompX: Explaining Transformers Decisions by Propagating Token Decompositioncited by 31
- 2024Interpreting Key Mechanisms of Factual Recall in Transformer-Based Language Modelscited by 30
- 2021The Case for Translation-Invariant Self-Attention in Transformer-Based Language Modelscited by 23
- 2024Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learningcited by 23
- 2024KernelSHAP-IQ: Weighted Least-Square Optimization for Shapley Interactionscited by 22
- 2020Recursive Non-Autoregressive Graph-to-Graph Transformer for Dependency Parsing with Iterative Refinementcited by 20
- 2024One-Shot Diffusion Mimicker for Handwritten Text Generationcited by 20
- 2021Effective Attention Sheds Light On Interpretabilitycited by 11
- 2025Beyond Isolated Words: Diffusion Brush for Handwritten Text-Line Generationcited by 10
- 2024Explaining Text Similarity in Transformer Modelscited by 9
- 2023Enhancing Cross-lingual Natural Language Inference by Soft Prompting with Multilingual Verbalizercited by 7
- 2023Zero-Shot Continuous Prompt Transfer: Generalizing Task Semantics Across Language Modelscited by 7
- 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Modelscited by 6
- 2024Cross-model Control: Improving Multiple Large Language Models in One-time Trainingcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.