mlsanityData to 7 Oct 2026

Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation

arXiv 2104.08771 ↗First posted 18 Apr 2021cs.CLcited by 74

Not on the map. Why?

Abstract (arXiv)

We study the power of cross-attention in the Transformer architecture within the context of transfer learning for machine translation, and extend the findings of studies into cross-attention when training from scratch. We conduct a series of experiments through fine-tuning a translation model on data where either the source or target language has changed. These experiments reveal that fine-tuning only the cross-attention parameters is nearly as effective as fine-tuning all parameters (i.e., the entire translation model). We provide insights into why this is the case and observe that limiting fine-tuning in this manner yields cross-lingually aligned embeddings. The implications of this finding for researchers and practitioners include a mitigation of catastrophic forgetting, the potential for zero-shot translation, and the ability to extend machine translation models to several new language pairs with reduced parameter storage overhead.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Builds on 32 of its 41 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

32 of 41 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  11. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  12. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
20 more
  1. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  2. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  3. 2019Are Sixteen Heads Really Better than One?cited by 803
  4. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  5. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  6. 2017Word Translation Without Parallel Datacited by 480
  7. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  8. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  9. 2016Transfer Learning for Low-Resource Neural Machine Translationcited by 188
  10. 2013Exploiting Similarities among Languages for Machine Translationcited by 187
  11. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  12. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 125
  13. 2018Rapid Adaptation of Neural Machine Translation to New Languagescited by 82
  14. 2017Transfer Learning across Low-Resource, Related Languages for Neural Machine Translationcited by 58
  15. 2018Trivial Transfer Learning for Low-Resource Neural Machine Translationcited by 47
  16. 2020Hard-Coded Gaussian Attention for Neural Machine Translationcited by 42
  17. 2020Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translationcited by 17
  18. 2019A Universal Parent Model for Low-Resource Neural Machine Translation Transfercited by 10
  19. 2020Not all parameters are born equal: Attention is mostly what you needcited by 6
  20. 2020Learning Contextualised Cross-lingual Word Embeddings and Alignments for Extremely Low-Resource Languages Using Parallel Corporacited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 12; 2024: 25; 2025: 19; 2026: 9.

74 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  2. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  3. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  4. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  5. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  6. 2023MA-SAM: Modality-agnostic SAM Adaptation for 3D Medical Image Segmentationcited by 82
  7. 2023IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuningcited by 45
  8. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  9. 2024Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Modelscited by 21
  10. 2024CMM-Math: A Chinese Multimodal Math Dataset To Evaluate and Enhance the Mathematics Reasoning of Large Multimodal Modelscited by 18
  11. 2022EtriCA: Event-Triggered Context-Aware Story Generation Augmented by Cross Attentioncited by 17
  12. 2023Language Models for German Text Simplification: Overcoming Parallel Data Scarcity through Style-specific Pre-trainingcited by 15
38 more
  1. 2023Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translationcited by 14
  2. 2024See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decompositioncited by 13
  3. 2025PLoP: Precise LoRA Placement for Efficient Finetuning of Large Modelscited by 13
  4. 2022UM4: Unified Multilingual Multiple Teacher-Student Model for Zero-Resource Neural Machine Translationcited by 10
  5. 2024Dynamic Layer Tying for Parameter-Efficient Transformerscited by 10
  6. 2024Controllable Prompt Tuning For Balancing Group Distributional Robustnesscited by 10
  7. 2025Parameter-Efficient Adaptation of Geospatial Foundation Models through Embedding Deflectioncited by 10
  8. 2024Robust Semi-Supervised Learning for Self-learning Open-World Classescited by 9
  9. 2024PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptationcited by 8
  10. 2025Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptationcited by 8
  11. 2022When does Parameter-Efficient Transfer Learning Work for Machine Translation?cited by 7
  12. 2023Cross Attention Transformers for Multi-modal Unsupervised Whole-Body PET Anomaly Detectioncited by 7
  13. 2023Neural Machine Translation for the Indigenous Languages of the Americas: An Introductioncited by 7
  14. 2024Block-wise LoRA: Revisiting Fine-grained LoRA for Effective Personalization and Stylization in Text-to-Image Generationcited by 7
  15. 2024LoRA-Guard: Parameter-Efficient Guardrail Adaptation for Content Moderation of Large Language Modelscited by 7
  16. 2025Orthogonal Finetuning Made Scalablecited by 7
  17. 2022HiMODE: A Hybrid Monocular Omnidirectional Depth Estimation Modelcited by 6
  18. 2022Continual Learning of Neural Machine Translation within Low Forgetting Risk Regionscited by 6
  19. 2024Light-VQA+: A Video Quality Assessment Model for Exposure Correction with Vision-Language Guidancecited by 6
  20. 2024Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuningcited by 6
  21. 2023A Cross-Attention Augmented Model for Event-Triggered Context-Aware Story Generationcited by 5
  22. 2024Investigating Neural Machine Translation for Low-Resource Languages: Using Bavarian as a Case Studycited by 5
  23. 2024SPD-CFL: Stepwise Parameter Dropout for Efficient Continual Federated Learningcited by 5
  24. 2022Know Where You're Going: Meta-Learning for Parameter-Efficient Fine-Tuningcited by 4
  25. 2024PriorCLIP: Visual Prior Guided Vision-Language Model for Remote Sensing Image-Text Retrievalcited by 4
  26. 2024Analyzing the Attention Heads for Pronoun Disambiguation in Context-aware Machine Translation Modelscited by 4
  27. 2025Memory-Efficient Fine-Tuning of Transformers via Token Selectioncited by 4
  28. 2023Parameter-Efficient Sparse Retrievers and Rerankers using Adapterscited by 3
  29. 2024FollowGen: A Scaled Noise Conditional Diffusion Model for Car-Following Trajectory Predictioncited by 3
  30. 2025Multitask Auxiliary Network for Perceptual Quality Assessment of Non-Uniformly Distorted Omnidirectional Imagescited by 3
  31. 2023Jointly Reparametrized Multi-Layer Adaptation for Efficient and Private Tuningcited by 2
  32. 2024A critical review of methods and challenges in large language modelscited by 2
  33. 2024Crossfusor: A Cross-Attention Transformer Enhanced Conditional Diffusion Model for Car-Following Trajectory Predictioncited by 2
  34. 2024Improving Low-Resource Machine Translation via Cross-Linguistic Transfer from Typologically Similar High-Resource Languagescited by 2
  35. 2025Generalizable Radio-Frequency Radiance Fields for Spatial Spectrum Synthesiscited by 2
  36. 2025M-SCAN: A Multistage Framework for Lumbar Spinal Canal Stenosis Grading Using Multi-View Cross Attentioncited by 2
  37. 2025Non-Stationary Time Series Forecasting Based on Fourier Analysis and Cross Attention Mechanismcited by 2
  38. 2026PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspectivecited by 2

50 most cited of 74 All 74 (JSON)