# Context-Aware Self-Attention Networks

arXiv: https://arxiv.org/abs/1902.05766 · HTML: https://mlsanity.com/paper/1902.05766/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Feb 2019.

## Built on by (most cited first, 25 of 25)

- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Prevent the Language Model from being Overconfident in Neural Machine Translation](https://mlsanity.com/paper/2105.11098/) (2021)
- [When did you become so smart, oh wise one?! Sarcasm Explanation in Multi-modal Multi-party Dialogues](https://mlsanity.com/paper/2203.06419/) (2022)
- [Attention Mechanism in Neural Networks: Where it Comes and Where it Goes](https://mlsanity.com/paper/2204.13154/) (2022)
- [Mask Attention Networks: Rethinking and Strengthen Transformer](https://mlsanity.com/paper/2103.13597/) (2021)
- [Auto-Parsing Network for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/2108.10568/) (2021)
- [Text Compression-aided Transformer Encoding](https://mlsanity.com/paper/2102.05951/) (2021)
- [MEMEX: Detecting Explanatory Evidence for Memes via Knowledge-Enriched Contextualization](https://mlsanity.com/paper/2305.15913/) (2023)
- [LMSOC: An Approach for Socially Sensitive Pretraining](https://mlsanity.com/paper/2110.10319/) (2021)
- [Explaining (Sarcastic) Utterances to Enhance Affect Understanding in Multimodal Dialogues](https://mlsanity.com/paper/2211.11049/) (2022)
- [Context-aware attention layers coupled with optimal transport domain adaptation and multimodal fusion methods for recognizing dementia from spontaneous speech](https://mlsanity.com/paper/2305.16406/) (2023)
- [SemEval 2024 -- Task 10: Emotion Discovery and Reasoning its Flip in Conversation (EDiReF)](https://mlsanity.com/paper/2402.18944/) (2024)
- [Context-Aware Interaction Network for Question Matching](https://mlsanity.com/paper/2104.08451/) (2021)
- [Constructing Phrase-level Semantic Labels to Form Multi-Grained Supervision for Image-Text Retrieval](https://mlsanity.com/paper/2109.05523/) (2021)
- [Discrete Auto-regressive Variational Attention Models for Text Modeling](https://mlsanity.com/paper/2106.08571/) (2021)
- [ANNA: Enhanced Language Representation for Question Answering](https://mlsanity.com/paper/2203.14507/) (2022)
- [From Multilingual Complexity to Emotional Clarity: Leveraging Commonsense to Unveil Emotions in Code-Mixed Dialogues](https://mlsanity.com/paper/2310.13080/) (2023)
- [Deps-SAN: Neural Machine Translation with Dependency-Scaled Self-Attention Network](https://mlsanity.com/paper/2111.11707/) (2021)
- [Tri-Attention: Explicit Context-Aware Attention Mechanism for Natural Language Processing](https://mlsanity.com/paper/2211.02899/) (2022)
- [Interaction is all You Need? A Study of Robots Ability to Understand and Execute](https://mlsanity.com/paper/2311.07150/) (2023)
- [Harmonizing Code-mixed Conversations: Personality-assisted Code-mixed Response Generation in Dialogues](https://mlsanity.com/paper/2401.12995/) (2024)
- [Attention Mechanism with Energy-Friendly Operations](https://mlsanity.com/paper/2204.13353/) (2022)
- [Towards Reliable Neural Machine Translation with Consistency-Aware Meta-Learning](https://mlsanity.com/paper/2303.10966/) (2023)
- [Injecting linguistic knowledge into BERT for Dialogue State Tracking](https://mlsanity.com/paper/2311.15623/) (2023)
- [DCS-ST for Classification of Breast Cancer Histopathology Images with Limited Annotations](https://mlsanity.com/paper/2505.03204/) (2025)

## Abstract (arXiv)

Self-attention model have shown its flexibility in parallel computation and the effectiveness on modeling both long- and short-term dependencies. However, it calculates the dependencies between representations without considering the contextual information, which have proven useful for modeling dependencies among neural representations in various natural language tasks. In this work, we focus on improving self-attention networks through capturing the richness of context. To maintain the simplicity and flexibility of the self-attention networks, we propose to contextualize the transformations of the query and key layers, which are used to calculates the relevance between elements. Specifically, we leverage the internal representations that embed both global and deep contexts, thus avoid relying on external resources. Experimental results on WMT14 English-German and WMT17 Chinese-English translation tasks demonstrate the effectiveness and universality of the proposed methods. Furthermore, we conducted extensive analyses to quantity how the context vectors participate in the self-attention model.