mlsanityData to 7 Oct 2026

Align-to-Distill: Trainable Attention Alignment for Knowledge Distillation in Neural Machine Translation

arXiv 2403.01479 ↗First posted 3 Mar 2024cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

The advent of scalable deep models and large datasets has improved the performance of Neural Machine Translation. Knowledge Distillation (KD) enhances efficiency by transferring knowledge from a teacher model to a more compact student model. However, KD approaches to Transformer architecture often rely on heuristics, particularly when deciding which teacher layers to distill from. In this paper, we introduce the 'Align-to-Distill' (A2D) strategy, designed to address the feature mapping problem by adaptively aligning student attention heads with their teacher counterparts during training. The Attention Alignment Module in A2D performs a dense head-by-head comparison between student and teacher attention heads across layers, turning the combinatorial mapping heuristics into a learning problem. Our experiments show the efficacy of A2D, demonstrating gains of up to +3.61 and +0.63 BLEU points for WMT-2022 De->Dsb and WMT-2014 En->De, respectively, compared to Transformer baselines.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 32 of its 38 references that are in the map. First posted on arXiv 3 Mar 2024.

Builds on

32 of 38 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  12. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
20 more
  1. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  2. 2020Knowledge Distillation: A Surveycited by 1,578
  3. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  4. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  5. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  7. 2016Sequence-Level Knowledge Distillationcited by 761
  8. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  9. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  10. 2013Do Deep Nets Really Need to be Deep?cited by 466
  11. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 171
  12. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 104
  13. 2016Learning to Translate in Real-time with Neural Machine Translationcited by 97
  14. 2020Pre-trained Summarization Distillationcited by 70
  15. 2020ALP-KD: Attention-Based Layer Projection for Knowledge Distillationcited by 65
  16. 2021Selective Knowledge Distillation for Neural Machine Translationcited by 42
  17. 2021Dynamic Knowledge Distillation for Pre-trained Language Modelscited by 38
  18. 2021Distilling Linguistic Context for Language Model Compressioncited by 33
  19. 2020Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layerscited by 24
  20. 2021Pay Better Attention to Attention: Head Selection in Multilingual and Multi-Domain Sequence Modelingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  2. 2025Scalable Model Merging with Progressive Layer-wise Distillationcited by 9
  3. 2025Attention Grounded Enhancement for Visual Document Retrievalcited by 3
  4. 2025IA2: Alignment with ICL Activations Improves Supervised Fine-Tuningcited by 1
  5. 2026Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoningcited by 0