# Context-Aware Cross-Attention for Non-Autoregressive Translation

arXiv: https://arxiv.org/abs/2011.00770 · HTML: https://mlsanity.com/paper/2011.00770/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Nov 2020.

## Built on by (most cited first, 26 of 26)

- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts](https://mlsanity.com/paper/2405.11804/) (2024)
- [Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation](https://mlsanity.com/paper/2106.00903/) (2021)
- [Learning Graph Neural Networks for Image Style Transfer](https://mlsanity.com/paper/2207.11681/) (2022)
- [Progressive Multi-Granularity Training for Non-Autoregressive Translation](https://mlsanity.com/paper/2106.05546/) (2021)
- [Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?](https://mlsanity.com/paper/2308.12898/) (2023)
- [E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generation](https://mlsanity.com/paper/2205.14912/) (2022)
- [Modeling Concentrated Cross-Attention for Neural Machine Translation with Gaussian Mixture Model](https://mlsanity.com/paper/2109.05244/) (2021)
- [Improving Neural Machine Translation by Bidirectional Training](https://mlsanity.com/paper/2109.07780/) (2021)
- [Knowledge Graph Augmented Network Towards Multiview Representation Learning for Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2201.04831/) (2022)
- [On the Complementarity between Pre-Training and Random-Initialization for Resource-Rich Machine Translation](https://mlsanity.com/paper/2209.03316/) (2022)
- [The USYD-JD Speech Translation System for IWSLT 2021](https://mlsanity.com/paper/2107.11572/) (2021)
- [Revisiting Token Dropping Strategy in Efficient BERT Pretraining](https://mlsanity.com/paper/2305.15273/) (2023)
- [Bridging the Gap Between Clean Data Training and Real-World Inference for Spoken Language Understanding](https://mlsanity.com/paper/2104.06393/) (2021)
- [Self-Guided Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/2105.04475/) (2021)
- [Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring](https://mlsanity.com/paper/2109.04411/) (2021)
- [Vega-MT: The JD Explore Academy Translation System for WMT22](https://mlsanity.com/paper/2209.09444/) (2022)
- [BLISS: Robust Sequence-to-Sequence Learning via Self-Supervised Input Representation](https://mlsanity.com/paper/2204.07837/) (2022)
- [Non-Autoregressive Neural Machine Translation: A Call for Clarity](https://mlsanity.com/paper/2205.10577/) (2022)
- [MirrorAlign: A Super Lightweight Unsupervised Word Alignment Model via Cross-Lingual Contrastive Learning](https://mlsanity.com/paper/2102.04009/) (2021)
- [Prompt-Learning for Cross-Lingual Relation Extraction](https://mlsanity.com/paper/2304.10354/) (2023)
- [Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing](https://mlsanity.com/paper/2506.13485/) (2025)
- [Original or Translated? On the Use of Parallel Data for Translation Quality Estimation](https://mlsanity.com/paper/2212.10257/) (2022)
- [OmniForce: On Human-Centered, Large Model Empowered and Cloud-Edge Collaborative AutoML System](https://mlsanity.com/paper/2303.00501/) (2023)
- [Revisiting Non-Autoregressive Translation at Scale](https://mlsanity.com/paper/2305.16155/) (2023)
- [Tri-Attention: Explicit Context-Aware Attention Mechanism for Natural Language Processing](https://mlsanity.com/paper/2211.02899/) (2022)

## Abstract (arXiv)

Non-autoregressive translation (NAT) significantly accelerates the inference process by predicting the entire target sequence. However, due to the lack of target dependency modelling in the decoder, the conditional generation process heavily depends on the cross-attention. In this paper, we reveal a localness perception problem in NAT cross-attention, for which it is difficult to adequately capture source context. To alleviate this problem, we propose to enhance signals of neighbour source tokens into conventional cross-attention. Experimental results on several representative datasets show that our approach can consistently improve translation quality over strong NAT baselines. Extensive analyses demonstrate that the enhanced cross-attention achieves better exploitation of source contexts by leveraging both local and global information.