Indirect Attention: Turning Context Misalignment into a Feature
Not on the map. Why?
Abstract (arXiv)
The attention mechanism has become a cornerstone of modern deep learning architectures, where keys and values are typically derived from the same underlying sequence or representation. This work explores a less conventional scenario, when keys and values originate from different sequences or modalities. Specifically, we first analyze the attention mechanism's behavior under noisy value features, establishing a critical noise threshold beyond which signal degradation becomes significant. Furthermore, we model context (key, value) misalignment as an effective form of structured noise within the value features, demonstrating that the noise induced by such misalignment can substantially exceed this critical threshold, thereby compromising standard attention's efficacy. Motivated by this, we introduce Indirect Attention, a modified attention mechanism that infers relevance indirectly in scenarios with misaligned context. We evaluate the performance of Indirect Attention across a range of synthetic tasks and real world applications, showcasing its superior ability to handle misalignment.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 39 references that are in the map. First posted on arXiv 30 Sep 2025.
Builds on
31 of 39 references.
References · 31
- 2017Attention Is All You Needcited by 59,932
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2013Intriguing properties of neural networkscited by 5,114
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
- 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
- 2022A Generalist Agentcited by 802
19 more
- 2016Fully-Convolutional Siamese Networks for Object Trackingcited by 566
- 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
- 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
- 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 206
- 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
- 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
- 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
- 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 84
- 2019One-Shot Object Detection with Co-Attention and Co-Excitationcited by 67
- 2022Meta-DETR: Image-Level Few-Shot Detection with Inter-Class Correlation Exploitationcited by 48
- 2018One-Shot Instance Segmentationcited by 43
- 2021Improved Text Classification via Contrastive Adversarial Trainingcited by 38
- 2022FS-DETR: Few-Shot DEtection TRansformer with prompting and without re-trainingcited by 20
- 2021Alignment Attention by Matching Key and Query Distributionscited by 14
- 2022Semantic-aligned Fusion Transformer for One-shot Object Detectioncited by 13
- 2024Benign Overfitting in Single-Head Attentioncited by 11
- 2021Making Attention Mechanisms More Robust and Interpretable with Virtual Adversarial Trainingcited by 8
- 2023Multimodal Transformer Using Cross-Channel attention for Object Detection in Remote Sensing Imagescited by 2
- 2023Exploration of visual prompt in Grounded pre-trained open-set detectioncited by 1
Built on by
Not cited yet.