Warranted Attention: Learning What to Pass from Attention to Prediction
Not on the map. Why?
Abstract (arXiv)
Relevance of information read by attention does not guarantee that its contribution benefits the current prediction. We propose Warrant, which learns how strongly attention-derived item contributions should be transmitted under the current query. Warrant applies learned item-wise permission before aggregation without renormalization, jointly controlling relative allocation and total transmission mass. Across backbones, we construct interfaces connecting these con- tributions to prediction scores or states and compare against ungated models on the same paths. On three CyGNet datasets, ungated paths reduce MRR, whereas learned permission partially or almost fully recovers the losses. In a 5-seed HotpotQA/RoBERTa experiment, distractors receive lower permission than gold support, while mean Support MRR rises from .9111 to .9141 and the unsupported selection rate falls from .2271 to .2215. Performance comparisons and contribution- level interventions across five task families reveal both the effects and limits of selective control. The results support learning the strength of contributions transmitted to prediction separately from attention relevance
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 43 of its 55 references that are in the map. First posted on arXiv 29 Jun 2026.
Builds on
43 of 55 references.
References · 43
- 2017Attention Is All You Needcited by 59,932
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2017On Calibration of Modern Neural Networkscited by 4,860
- 2017Squeeze-and-Excitation Networkscited by 4,763
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2020Longformer: The Long-Document Transformercited by 3,362
- 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
- 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
31 more
- 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
- 2019Attention is not Explanationcited by 916
- 2017Selective Classification for Deep Neural Networkscited by 799
- 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
- 2016Language Modeling with Gated Convolutional Networkscited by 688
- 2019Attention is not not Explanationcited by 629
- 2017Deep Interest Network for Click-Through Rate Predictioncited by 541
- 2020Inductive Representation Learning on Temporal Graphscited by 454
- 2019Predicting Dynamic Embedding Trajectory in Temporal Interaction Networkscited by 443
- 2019Is Attention Interpretable?cited by 387
- 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355
- 2017Residual Gated Graph ConvNetscited by 341
- 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
- 2015Highway Networkscited by 257
- 2022Mega: Moving Average Equipped Gated Attentioncited by 178
- 2023Towards Better Dynamic Graph Learning: New Architecture and Unified Librarycited by 154
- 2018Learning Sequence Encoders for Temporal Knowledge Graph Completioncited by 145
- 2023Do We Really Need Complicated Model Architectures For Temporal Networks?cited by 141
- 2021Global Self-Attention as a Replacement for Graph Convolutioncited by 113
- 2020Neural Spatio-Temporal Point Processescited by 92
- 2021Transformer Embeddings of Irregularly Spaced Events and Their Participantscited by 90
- 2020Learning from History: Modeling Temporal Knowledge Graphs with Sequential Copy-Generation Networkscited by 84
- 2016Improving Neural Language Models with a Continuous Cachecited by 81
- 2024Theory, Analysis, and Best Practices for Sigmoid Self-Attentioncited by 62
- 2021Adaptive Nearest Neighbor Machine Translationcited by 58
- 2023Quantifying Context Mixing in Transformerscited by 43
- 2022Temporal Knowledge Graph Reasoning with Historical Contrastive Learningcited by 42
- 2021Evidentiality-guided Generation for Knowledge-Intensive NLP Taskscited by 40
- 2016Gated-Attention Readers for Text Comprehensioncited by 37
- 2022Towards Robust k-Nearest-Neighbor Machine Translationcited by 15
- 2020xERTE: Explainable Reasoning on Temporal Knowledge Graphs for Forecasting Future Linkscited by 13
Built on by
Not cited yet.